Sep 09 21:33:14.388064 ip-10-0-133-204 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 21:33:14.819430 ip-10-0-133-204 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:33:14.819430 ip-10-0-133-204 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 21:33:14.819430 ip-10-0-133-204 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:33:14.819430 ip-10-0-133-204 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 21:33:14.819430 ip-10-0-133-204 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:33:14.821635 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.821549    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 21:33:14.825083 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825067    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:14.825083 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825084    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825089    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825092    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825095    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825098    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825101    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825104    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825114    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825118    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825120    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825123    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825126    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825128    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825131    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825133    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825136    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825139    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825142    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825144    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825147    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:14.825148 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825150    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825152    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825155    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825158    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825161    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825164    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825166    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825169    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825171    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825174    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825177    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825179    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825181    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825184    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825186    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825189    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825191    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825193    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825197    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825200    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:14.825624 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825202    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825204    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825207    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825209    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825212    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825214    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825218    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825222    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825224    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825227    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825230    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825233    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825236    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825239    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825242    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825245    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825247    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825250    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825252    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:14.826123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825255    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825257    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825260    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825264    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825267    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825270    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825273    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825276    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825278    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825281    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825283    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825286    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825289    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825292    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825294    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825297    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825300    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825302    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825305    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:14.826581 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825307    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825311    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825314    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825316    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825319    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825322    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825324    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825328    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825331    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825333    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825336    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825338    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825341    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825343    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825346    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825348    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825351    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825354    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825358    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.825361    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:14.827055 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826112    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826122    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826126    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826129    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826132    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826135    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826139    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826142    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826145    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826148    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826150    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826153    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826155    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826158    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826161    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826163    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826168    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826171    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826174    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:14.827537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826178    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826181    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826184    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826187    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826190    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826192    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826195    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826197    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826202    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826206    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826209    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826212    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826214    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826217    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826219    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826222    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826224    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826227    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826229    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:14.828030 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826232    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826235    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826237    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826240    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826242    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826245    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826247    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826250    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826252    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826255    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826258    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826261    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826263    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826266    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826269    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826272    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826274    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826276    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826279    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826281    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:14.828504 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826284    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826286    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826289    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826291    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826293    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826296    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826298    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826301    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826303    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826305    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826308    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826311    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826324    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826328    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826332    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826336    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826338    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826341    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826344    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826346    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:14.829014 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826349    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826351    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826354    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826357    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826360    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826363    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826366    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826369    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826371    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826374    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826376    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826379    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826381    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826384    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826387    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826389    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826392    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826394    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826397    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826399    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:14.829498 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826401    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826794    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826799    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826802    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826805    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826807    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826811    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826814    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826817    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826819    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826822    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826825    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826827    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826830    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826833    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826835    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826838    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826841    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826844    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826847    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:14.830091 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826849    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826852    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826855    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826857    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826860    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826862    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826865    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826867    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826870    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826873    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826877    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826881    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826885    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826889    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826891    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826894    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826896    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826899    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826901    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826905    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:14.830593 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826907    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826910    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826912    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826915    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826917    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826920    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826923    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826926    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826928    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826931    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826934    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826936    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826939    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826941    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826944    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826946    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826949    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826952    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826954    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:14.831095 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826957    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826959    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826962    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826964    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826966    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826969    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826973    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826975    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826978    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826980    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826984    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826987    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826990    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826993    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826996    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.826998    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827001    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827004    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827006    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827012    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:14.831558 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827016    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827018    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827021    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827025    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827027    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827030    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827032    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827035    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827037    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827040    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827042    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827045    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827047    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827050    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827052    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827055    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827058    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827062    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827065    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:14.832102 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827068    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.827071    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827179    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827210    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827217    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827221    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827225    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827229    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827234    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827238    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827241    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827244    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827250    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827253    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827256    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827260    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827262    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827265    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827268    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827271    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827683    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827687    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827691    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827694    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 21:33:14.832557 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827698    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827702    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827705    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827708    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827728    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827733    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827738    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827742    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827746    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827749    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827753    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827756    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827759    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827762    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827764    2563 flags.go:64] FLAG: --enable-server="true"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827768    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827774    2563 flags.go:64] FLAG: --event-burst="100"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827777    2563 flags.go:64] FLAG: --event-qps="50"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827779    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827782    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827788    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827792    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827795    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827798    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827801    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 09 21:33:14.833179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827804    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827807    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827809    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827812    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827815    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827818    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827820    2563 flags.go:64] FLAG: --feature-gates=""
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827824    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827827    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827830    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827833    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827837    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827840    2563 flags.go:64] FLAG: --help="false"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827843    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-204.ec2.internal"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827846    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827849    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827852    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827855    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827859    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827862    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827864    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827867    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827870    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 21:33:14.833810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827873    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827876    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827879    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827882    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827886    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827889    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827899    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827902    2563 flags.go:64] FLAG: --lock-file=""
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827905    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827908    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827911    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827917    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827919    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827922    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827925    2563 flags.go:64] FLAG: --logging-format="text"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827928    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827931    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827934    2563 flags.go:64] FLAG: --manifest-url=""
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827937    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827941    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827946    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827950    2563 flags.go:64] FLAG: --max-pods="110"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827953    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827956    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827959    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 21:33:14.834383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827962    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827964    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827967    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827970    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827981    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827985    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827988    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827991    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827993    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.827999    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828002    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828005    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828009    2563 flags.go:64] FLAG: --port="10250"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828012    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828015    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0778bb2c06fafe448"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828018    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828021    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828024    2563 flags.go:64] FLAG: --register-node="true"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828027    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828030    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828033    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828037    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828039    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828043    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828046    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 21:33:14.835172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828049    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828052    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828056    2563 flags.go:64] FLAG: --runonce="false"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828058    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828061    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828064    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828067    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828070    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828073    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828076    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828079    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828082    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828084    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828087    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828090    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828093    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828096    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828103    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828108    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828112    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828115    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828119    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828122    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828125    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828128    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 21:33:14.836207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828131    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828134    2563 flags.go:64] FLAG: --v="2"
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828138    2563 flags.go:64] FLAG: --version="false"
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828142    2563 flags.go:64] FLAG: --vmodule=""
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828149    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.828152    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828273    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828276    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828279    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828283    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828287    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828290    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828293    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828296    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828298    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828301    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828304    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828306    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828309    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828312    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828314    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:14.837068 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828317    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828319    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828322    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828324    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828328    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828331    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828334    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828337    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828339    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828341    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828344    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828346    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828349    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828351    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828354    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828356    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828359    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828361    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828364    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828366    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:14.837618 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828370    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828373    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828375    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828378    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828380    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828383    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828385    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828388    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828391    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828395    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828398    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828400    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828403    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828405    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828408    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828410    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828412    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828419    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828421    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:14.838132 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828424    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828426    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828429    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828431    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828434    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828436    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828439    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828441    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828444    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828446    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828449    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828451    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828453    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828458    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828460    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828463    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828465    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828468    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828470    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828473    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:14.838600 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828476    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828478    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828480    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828484    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828488    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828491    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828494    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828496    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828499    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828501    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828505    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828509    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828512    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828516    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828519    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828522    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828525    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828527    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:14.839105 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828530    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828532    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828534    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828537    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828540    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828542    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.828544    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.829080    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.838190    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.838206    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.839008    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.839017    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839080    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839085    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839089    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:14.839539 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839091    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839094    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839097    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839100    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839103    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839106    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839109    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839112    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839115    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839118    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839121    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839125    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839127    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839130    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839132    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839135    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839138    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839140    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839144    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:14.839941 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839146    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839149    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839152    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839155    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839157    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839160    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839163    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839165    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839168    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839170    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839173    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839176    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839179    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839184    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839187    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839189    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839191    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839194    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839197    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:14.840403 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839200    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839203    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839206    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839209    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839212    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839215    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839217    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839220    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839223    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839225    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839227    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839230    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839232    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839235    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839237    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839240    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839242    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839245    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839247    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839250    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:14.840885 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839252    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839255    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839258    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839260    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839263    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839265    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839268    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839270    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839272    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839275    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839277    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839280    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839282    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839284    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839287    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839289    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839292    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839294    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839303    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839305    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:14.841390 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839308    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839310    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839313    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839315    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839317    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839320    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839323    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839325    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839328    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839330    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839333    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839335    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839338    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839342    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839346    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839349    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839352    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:14.841963 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839354    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.839359    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839477    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839482    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839485    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839487    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839490    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839493    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839496    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839499    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839501    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839504    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839507    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839509    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839517    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:33:14.842374 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839520    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839522    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839525    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839528    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839530    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839533    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839535    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839538    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839540    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839543    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839545    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839548    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839551    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839553    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839556    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839558    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839561    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839563    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839566    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839568    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:33:14.842760 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839570    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839573    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839575    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839578    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839580    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839583    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839585    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839588    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839590    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839593    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839595    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839597    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839601    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839603    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839606    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839609    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839612    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839615    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839617    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:33:14.843234 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839620    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839623    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839627    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839630    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839634    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839636    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839639    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839642    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839644    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839647    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839650    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839653    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839657    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839660    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839663    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839666    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839669    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839671    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839674    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:33:14.843685 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839676    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839679    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839681    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839684    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839686    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839688    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839691    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839694    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839696    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839699    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839701    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839704    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839706    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839708    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839711    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839729    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839732    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839735    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839738    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:33:14.844156 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839740    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839744    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839747    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839749    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839752    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839754    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839757    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839759    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:14.839761    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.839766    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.839888    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.841931    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.843076    2563 server.go:1019] "Starting client certificate rotation"
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.843181    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 21:33:14.844613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.843216    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 21:33:14.845607 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.845592    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 21:33:14.847288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.847269    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 21:33:14.857174 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.857153    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 09 21:33:14.861094 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.860942    2563 log.go:25] "Validated CRI v1 image API"
Sep 09 21:33:14.862351 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.862332    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 21:33:14.866021 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.866002    2563 fs.go:135] Filesystem UUIDs: map[1e8e16c7-8910-43d6-b51f-bc671e3d70b9:/dev/nvme0n1p3 363de5db-a7e5-47da-aa42-12729bad883f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 21:33:14.866124 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.866022    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 21:33:14.873196 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.873170    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 21:33:14.873320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.872872    2563 manager.go:216] Machine: {Timestamp:2026-09-09 21:33:14.870140312 +0000 UTC m=+0.356023341 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095911 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2825f24c2e1e45e0e50aa733e3d251 SystemUUID:ec2825f2-4c2e-1e45-e0e5-0aa733e3d251 BootID:d64c9a21-643a-45dc-b737-5f5967010ee8 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9a:d2:3b:65:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9a:d2:3b:65:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:73:58:80:ea:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 21:33:14.873320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.873219    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 21:33:14.873415 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.873378    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 21:33:14.876166 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.876141    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 21:33:14.876306 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.876171    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 21:33:14.876965 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.876954    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 21:33:14.877003 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.876969    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 21:33:14.877003 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.876976    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 21:33:14.877959 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.877948    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 21:33:14.879158 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.879146    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 21:33:14.882203 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.882191    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 21:33:14.882333 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.882324    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 21:33:14.882366 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.882359    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 21:33:14.882394 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.882384    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 21:33:14.882421 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.882408    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 21:33:14.882448 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.882422    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 09 21:33:14.882448 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.882436    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 21:33:14.883040 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.883028    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 21:33:14.883082 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.883047    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 21:33:14.885378 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.885358    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 21:33:14.887505 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.887487    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 21:33:14.887605 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.887526    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 21:33:14.888275 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888260    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 21:33:14.888347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888283    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 21:33:14.888347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888292    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 21:33:14.888347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888300    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 21:33:14.888347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888319    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 21:33:14.888347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888328    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 21:33:14.888347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888346    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 21:33:14.888547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888355    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 21:33:14.888547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888366    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 21:33:14.888547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888374    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 21:33:14.888547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888387    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 21:33:14.888547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.888396    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 21:33:14.889856 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.889827    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 21:33:14.889856 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.889858    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 21:33:14.893984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.893966    2563 server.go:1268] "Started kubelet"
Sep 09 21:33:14.894102 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.894048    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 21:33:14.894171 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.894106    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 21:33:14.894220 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.894191    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 21:33:14.894405 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.894385    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-204.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 21:33:14.894534 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.894479    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 21:33:14.894930 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.894637    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 21:33:14.895096 ip-10-0-133-204 systemd[1]: Started Kubernetes Kubelet.
Sep 09 21:33:14.895392 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.895376    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 21:33:14.895918 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.895798    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 21:33:14.897091 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.897072    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tsxvx"
Sep 09 21:33:14.898578 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.898561    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 21:33:14.899161 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.899143    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 21:33:14.899871 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.899857    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 21:33:14.899970 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.899951    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:14.900454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.900437    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 21:33:14.900530 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.900440    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 21:33:14.900530 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.900523    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 21:33:14.900629 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.900603    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 21:33:14.900629 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.900612    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 21:33:14.900629 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.900620    2563 factory.go:55] Registering systemd factory
Sep 09 21:33:14.900761 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.900637    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 09 21:33:14.901019 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.901005    2563 factory.go:153] Registering CRI-O factory
Sep 09 21:33:14.901083 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.901022    2563 factory.go:223] Registration of the crio container factory successfully
Sep 09 21:33:14.901157 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.901145    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 21:33:14.901200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.901169    2563 factory.go:103] Registering Raw factory
Sep 09 21:33:14.901200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.901187    2563 manager.go:1196] Started watching for new ooms in manager
Sep 09 21:33:14.901610 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.901597    2563 manager.go:349] Starting recovery of all containers
Sep 09 21:33:14.903011 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.901796    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-204.ec2.internal.18d3c48ede859cd0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-204.ec2.internal,UID:ip-10-0-133-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-204.ec2.internal,},FirstTimestamp:2026-09-09 21:33:14.893917392 +0000 UTC m=+0.379800421,LastTimestamp:2026-09-09 21:33:14.893917392 +0000 UTC m=+0.379800421,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-204.ec2.internal,}"
Sep 09 21:33:14.903108 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.903091    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 21:33:14.904670 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.904642    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tsxvx"
Sep 09 21:33:14.908835 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.908792    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 21:33:14.913951 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.913933    2563 manager.go:354] Recovery completed
Sep 09 21:33:14.918602 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.918589    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:14.920952 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.920933    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:14.921035 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.920971    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:14.921035 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.920986    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:14.921607 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.921592    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 21:33:14.921607 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.921606    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 21:33:14.921773 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.921610    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:14.921773 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.921624    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 21:33:14.922384 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.922361    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:14.922476 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.922404    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:14.922476 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.922421    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:14.922709 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.922697    2563 policy_none.go:49] "None policy: Start"
Sep 09 21:33:14.923150 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.923140    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 21:33:14.923185 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.923155    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 21:33:14.924073 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.924063    2563 policy_none.go:47] "Start"
Sep 09 21:33:14.966430 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.966408    2563 manager.go:337] "Starting Device Plugin manager"
Sep 09 21:33:14.967265 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.966457    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 21:33:14.967265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.966471    2563 server.go:87] "Starting device plugin registration server"
Sep 09 21:33:14.967265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.966809    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 21:33:14.967265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.966823    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 21:33:14.967265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.966913    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 21:33:14.967265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.966998    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 21:33:14.967265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:14.967006    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 21:33:14.967670 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.967649    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 21:33:14.967782 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:14.967707    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.038114 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.038065    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 21:33:15.039427 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.039409    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 21:33:15.039507 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.039472    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 21:33:15.039507 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.039502    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 21:33:15.039596 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.039550    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 21:33:15.042220 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.042201    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:15.067089 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.067055    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:15.068117 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.068095    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:15.068239 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.068132    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:15.068239 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.068146    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:15.068239 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.068184    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.074227 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.074169    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.074227 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.074197    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-204.ec2.internal\": node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.087117 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.087085    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.139832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.139782    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"]
Sep 09 21:33:15.140033 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.139877    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:15.140888 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.140871    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:15.140983 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.140901    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:15.140983 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.140914    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:15.142323 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.142311    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:15.142497 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.142483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.142532 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.142517    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:15.143672 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.143654    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:15.143758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.143683    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:15.143758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.143693    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:15.143758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.143665    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:15.143847 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.143762    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:15.143847 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.143796    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:15.145279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.145265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.145332 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.145291    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:33:15.145981 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.145963    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:33:15.146080 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.145997    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:33:15.146080 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.146011    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:33:15.174565 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.174541    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-204.ec2.internal\" not found" node="ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.178885 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.178869    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-204.ec2.internal\" not found" node="ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.188091 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.188068    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.202496 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.202469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94d8dd3914b89c4faa93d8b0e772fd80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"94d8dd3914b89c4faa93d8b0e772fd80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.202574 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.202500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94d8dd3914b89c4faa93d8b0e772fd80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"94d8dd3914b89c4faa93d8b0e772fd80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.202574 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.202518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd5a5b93ecd8465e35800ce095df071a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.ec2.internal\" (UID: \"bd5a5b93ecd8465e35800ce095df071a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.288855 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.288828    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.303217 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.303189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94d8dd3914b89c4faa93d8b0e772fd80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"94d8dd3914b89c4faa93d8b0e772fd80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.303276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.303220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94d8dd3914b89c4faa93d8b0e772fd80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"94d8dd3914b89c4faa93d8b0e772fd80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.303276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.303238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd5a5b93ecd8465e35800ce095df071a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.ec2.internal\" (UID: \"bd5a5b93ecd8465e35800ce095df071a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.303276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.303266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd5a5b93ecd8465e35800ce095df071a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.ec2.internal\" (UID: \"bd5a5b93ecd8465e35800ce095df071a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.303383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.303283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94d8dd3914b89c4faa93d8b0e772fd80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"94d8dd3914b89c4faa93d8b0e772fd80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.303383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.303296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94d8dd3914b89c4faa93d8b0e772fd80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"94d8dd3914b89c4faa93d8b0e772fd80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.389654 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.389579    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.477152 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.477120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.481390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.481034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:15.490465 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.490443    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.591012 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.590972    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.691591 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.691510    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.739351 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.739320    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:15.792290 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.792257    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.842851 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.842809    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 21:33:15.843284 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.842987    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 21:33:15.843284 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.843006    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 21:33:15.893421 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.893383    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:15.907730 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.907666    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 21:28:15 +0000 UTC" deadline="2028-02-16 18:35:54.464650301 +0000 UTC"
Sep 09 21:33:15.907827 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.907735    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12597h2m38.556920471s"
Sep 09 21:33:15.960979 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:15.960796    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:15.994526 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:15.994462    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:16.095684 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:16.095660    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:16.104186 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:16.104153    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d8dd3914b89c4faa93d8b0e772fd80.slice/crio-2912401ea65730e5115d76ce7ae90d72b2ac197be6568ba633b1b0fe8ba982b2 WatchSource:0}: Error finding container 2912401ea65730e5115d76ce7ae90d72b2ac197be6568ba633b1b0fe8ba982b2: Status 404 returned error can't find the container with id 2912401ea65730e5115d76ce7ae90d72b2ac197be6568ba633b1b0fe8ba982b2
Sep 09 21:33:16.108715 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.108701    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:33:16.198193 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:16.198140    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:16.305657 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:16.305564    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Sep 09 21:33:16.356152 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.356121    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:16.400710 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.400680    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:16.411862 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.411834    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 21:33:16.413012 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.412998    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Sep 09 21:33:16.421381 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.421358    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 21:33:16.783996 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.783875    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:16.883648 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.883612    2563 apiserver.go:52] "Watching apiserver"
Sep 09 21:33:16.892788 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.892308    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 21:33:16.894084 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.894046    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7dlnc","openshift-dns/node-resolver-lrtz6","openshift-image-registry/node-ca-klgv6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal","openshift-multus/multus-additional-cni-plugins-hx62l","openshift-network-diagnostics/network-check-target-7qj82","openshift-network-operator/iptables-alerter-kvs2d","openshift-ovn-kubernetes/ovnkube-node-gs2lz","kube-system/konnectivity-agent-mnggj","kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q","openshift-cluster-node-tuning-operator/tuned-k44jm","openshift-multus/multus-c56fx","openshift-multus/network-metrics-daemon-h2p64"]
Sep 09 21:33:16.897245 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.897219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.899173 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.899149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:16.901817 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.900017    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 21:33:16.901817 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.900327    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 21:33:16.901817 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.900424    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 21:33:16.901817 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.900563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qcxtt\""
Sep 09 21:33:16.901817 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.900723    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 21:33:16.901817 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.901531    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 21:33:16.902415 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.902383    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 21:33:16.902542 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.902426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:16.902542 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.902441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dwrml\""
Sep 09 21:33:16.902542 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.902390    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 21:33:16.905389 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.904632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:16.905389 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:16.904716    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:16.905389 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.905263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 21:33:16.905389 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.905271    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kfp2c\""
Sep 09 21:33:16.905651 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.905414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 21:33:16.906136 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.905914    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 21:33:16.907875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.907319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:16.909714 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.909658    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:16.909873 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:16.909771    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:16.910610 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.910182    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 21:33:16.910610 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.910430    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8cbxc\""
Sep 09 21:33:16.910610 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.910511    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.911958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-conf-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.911993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-etc-kubernetes\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndv7z\" (UniqueName: \"kubernetes.io/projected/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-kube-api-access-ndv7z\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-cni-bin\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-hostroot\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvbg\" (UniqueName: \"kubernetes.io/projected/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-kube-api-access-jjvbg\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c59f67-3ceb-4cf5-a041-22852d768ac3-host\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-cni-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-cnibin\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-socket-dir-parent\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-netns\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-cni-multus\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-daemon-config\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-multus-certs\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-kubelet\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-hosts-file\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:16.912781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-tmp-dir\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:16.913763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3c59f67-3ceb-4cf5-a041-22852d768ac3-serviceca\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:16.913763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpz4\" (UniqueName: \"kubernetes.io/projected/f3c59f67-3ceb-4cf5-a041-22852d768ac3-kube-api-access-dvpz4\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:16.913763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-system-cni-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.913763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-cni-binary-copy\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.913763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-k8s-cni-cncf-io\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:16.913763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.912583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:16.917753 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.916671    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 21:33:16.917753 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.916687    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:33:16.917753 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.916707    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 21:33:16.917753 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.916914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:16.918001 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.917770    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kv8vh\""
Sep 09 21:33:16.918001 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.917987    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 21:33:16.920919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.920186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:16.922365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.921491    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 21:33:16.922365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.921523    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 21:33:16.922365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.921625    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 21:33:16.922365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.921718    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 21:33:16.922365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.921625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mxj4t\""
Sep 09 21:33:16.922365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.922134    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 21:33:16.922365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.922177    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 21:33:16.923960 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.923827    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 21:33:16.923960 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.923850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9zwlx\""
Sep 09 21:33:16.928679 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.924274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 21:33:16.931199 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.931178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:16.931316 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:16.931296    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:16.933589 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.933570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:16.936040 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.936019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:16.936286 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.936266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5f9ff\""
Sep 09 21:33:16.936414 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.936396    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 21:33:16.936476 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.936431    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 21:33:16.936534 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.936524    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 21:33:16.937263 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.937244    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-crmqv"
Sep 09 21:33:16.939779 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.939758    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6cn4p\""
Sep 09 21:33:16.939895 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.939860    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:33:16.940196 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.940176    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 21:33:16.944698 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:16.944677    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-crmqv"
Sep 09 21:33:17.001594 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.001569    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 21:33:17.013701 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-cni-multus\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.013881 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-daemon-config\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.013881 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-cni-multus\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.013881 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-device-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.014037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.014037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpz4\" (UniqueName: \"kubernetes.io/projected/f3c59f67-3ceb-4cf5-a041-22852d768ac3-kube-api-access-dvpz4\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:17.014037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysctl-conf\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.014037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-systemd\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.014037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-var-lib-kubelet\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.014037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.013999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-node-log\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-cni-netd\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-conf-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.014253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-kubernetes\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.014253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-kubelet-config\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.014253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-var-lib-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-cni-bin\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2snv\" (UniqueName: \"kubernetes.io/projected/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-kube-api-access-s2snv\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7d2\" (UniqueName: \"kubernetes.io/projected/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-kube-api-access-hz7d2\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-cni-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-socket-dir-parent\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysctl-d\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-systemd\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-ovn\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-registration-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.014501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-iptables-alerter-script\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-modprobe-d\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-log-socket\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfb42f63-665c-4bd7-bf86-233e503c3533-konnectivity-ca\") pod \"konnectivity-agent-mnggj\" (UID: \"cfb42f63-665c-4bd7-bf86-233e503c3533\") " pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgv7j\" (UniqueName: \"kubernetes.io/projected/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-kube-api-access-rgv7j\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-k8s-cni-cncf-io\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-host\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovnkube-script-lib\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-conf-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfjd\" (UniqueName: \"kubernetes.io/projected/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-kube-api-access-zzfjd\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-cni-bin\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.014949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-cni-binary-copy\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.014975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-cnibin\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-netns\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-multus-certs\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-daemon-config\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-run\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-dbus\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-cni-bin\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-run-netns\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovn-node-metrics-cert\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfb42f63-665c-4bd7-bf86-233e503c3533-agent-certs\") pod \"konnectivity-agent-mnggj\" (UID: \"cfb42f63-665c-4bd7-bf86-233e503c3533\") " pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-tmp-dir\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:17.015358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-tmp\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-cni-binary-copy\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndv7z\" (UniqueName: \"kubernetes.io/projected/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-kube-api-access-ndv7z\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-system-cni-dir\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-k8s-cni-cncf-io\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-netns\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-run-multus-certs\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-cnibin\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-socket-dir-parent\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.015875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-multus-cni-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-hostroot\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-tmp-dir\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-hostroot\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.015993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovnkube-config\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-env-overrides\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-cnibin\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8gmg\" (UniqueName: \"kubernetes.io/projected/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-kube-api-access-k8gmg\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3c59f67-3ceb-4cf5-a041-22852d768ac3-serviceca\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-cni-binary-copy\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.016320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-kubelet\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-hosts-file\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-host-var-lib-kubelet\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-lib-modules\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-hosts-file\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3c59f67-3ceb-4cf5-a041-22852d768ac3-serviceca\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-sys-fs\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.016829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-system-cni-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-etc-kubernetes\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvbg\" (UniqueName: \"kubernetes.io/projected/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-kube-api-access-jjvbg\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c59f67-3ceb-4cf5-a041-22852d768ac3-host\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-sys\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-etc-kubernetes\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-system-cni-dir\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c59f67-3ceb-4cf5-a041-22852d768ac3-host\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.016951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-systemd-units\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qnj\" (UniqueName: \"kubernetes.io/projected/0555c16f-7957-4472-a0bf-b9514e19ada1-kube-api-access-v9qnj\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysconfig\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-tuned\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-kubelet\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-slash\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-etc-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-socket-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.017341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-host-slash\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.018116 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.017356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:17.021551 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.021521    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 21:33:17.026799 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.026770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndv7z\" (UniqueName: \"kubernetes.io/projected/cc9fc8d2-e03f-44e4-89c2-2e6e697f8970-kube-api-access-ndv7z\") pod \"multus-c56fx\" (UID: \"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970\") " pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.027965 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.027916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvbg\" (UniqueName: \"kubernetes.io/projected/fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6-kube-api-access-jjvbg\") pod \"node-resolver-lrtz6\" (UID: \"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6\") " pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:17.029153 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.029127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpz4\" (UniqueName: \"kubernetes.io/projected/f3c59f67-3ceb-4cf5-a041-22852d768ac3-kube-api-access-dvpz4\") pod \"node-ca-klgv6\" (UID: \"f3c59f67-3ceb-4cf5-a041-22852d768ac3\") " pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:17.046609 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.046491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal" event={"ID":"bd5a5b93ecd8465e35800ce095df071a","Type":"ContainerStarted","Data":"12b3d6b6f45189a1ec8aaa6150513d1b4913a07f64ba5bd3ee7ea2787418bf76"}
Sep 09 21:33:17.049510 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.049475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal" event={"ID":"94d8dd3914b89c4faa93d8b0e772fd80","Type":"ContainerStarted","Data":"2912401ea65730e5115d76ce7ae90d72b2ac197be6568ba633b1b0fe8ba982b2"}
Sep 09 21:33:17.117649 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-registration-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.117649 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-iptables-alerter-script\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-modprobe-d\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-log-socket\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfb42f63-665c-4bd7-bf86-233e503c3533-konnectivity-ca\") pod \"konnectivity-agent-mnggj\" (UID: \"cfb42f63-665c-4bd7-bf86-233e503c3533\") " pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgv7j\" (UniqueName: \"kubernetes.io/projected/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-kube-api-access-rgv7j\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-host\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovnkube-script-lib\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfjd\" (UniqueName: \"kubernetes.io/projected/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-kube-api-access-zzfjd\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:17.117938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-cni-binary-copy\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-run\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-dbus\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.117995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-run-netns\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovn-node-metrics-cert\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfb42f63-665c-4bd7-bf86-233e503c3533-agent-certs\") pod \"konnectivity-agent-mnggj\" (UID: \"cfb42f63-665c-4bd7-bf86-233e503c3533\") " pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-tmp\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-system-cni-dir\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovnkube-config\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-env-overrides\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-cnibin\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8gmg\" (UniqueName: \"kubernetes.io/projected/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-kube-api-access-k8gmg\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-lib-modules\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.118454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-sys-fs\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-sys\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-systemd-units\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qnj\" (UniqueName: \"kubernetes.io/projected/0555c16f-7957-4472-a0bf-b9514e19ada1-kube-api-access-v9qnj\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysconfig\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-tuned\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-kubelet\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-slash\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-slash\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-modprobe-d\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-log-socket\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-sys-fs\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-registration-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.119270 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-run-netns\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-cni-binary-copy\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.118991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-run\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-sys\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-systemd-units\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-host\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfb42f63-665c-4bd7-bf86-233e503c3533-konnectivity-ca\") pod \"konnectivity-agent-mnggj\" (UID: \"cfb42f63-665c-4bd7-bf86-233e503c3533\") " pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysconfig\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-cnibin\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovnkube-config\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovnkube-script-lib\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.120004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.119905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0555c16f-7957-4472-a0bf-b9514e19ada1-system-cni-dir\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-iptables-alerter-script\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-env-overrides\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-kubelet\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-dbus\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-etc-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-socket-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-host-slash\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.120238    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-device-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-lib-modules\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.120332    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret podName:f85f5db0-2786-426b-a8dc-b6f7d6f37fed nodeName:}" failed. No retries permitted until 2026-09-09 21:33:17.620288737 +0000 UTC m=+3.106171784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret") pod "global-pull-secret-syncer-7dlnc" (UID: "f85f5db0-2786-426b-a8dc-b6f7d6f37fed") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-device-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysctl-conf\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-systemd\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.120645 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-var-lib-kubelet\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-node-log\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-cni-netd\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-kubernetes\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-kubelet-config\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-host-slash\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-var-lib-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-socket-dir\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-cni-bin\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-cni-bin\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.120653    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2snv\" (UniqueName: \"kubernetes.io/projected/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-kube-api-access-s2snv\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.120708    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs podName:0ae5a6d6-2f5b-4af6-ab7a-b629addbf014 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:17.620693805 +0000 UTC m=+3.106576835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs") pod "network-metrics-daemon-h2p64" (UID: "0ae5a6d6-2f5b-4af6-ab7a-b629addbf014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7d2\" (UniqueName: \"kubernetes.io/projected/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-kube-api-access-hz7d2\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysctl-d\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.121578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-systemd\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-etc-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-ovn\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-kubernetes\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-systemd\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-ovn\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-node-log\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-var-lib-openvswitch\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-cni-netd\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysctl-d\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-run-systemd\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.120986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-kubelet-config\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-var-lib-kubelet\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0555c16f-7957-4472-a0bf-b9514e19ada1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.121834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-sysctl-conf\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.122408 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.122127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-ovn-node-metrics-cert\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.123185 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.122190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-etc-tuned\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.123185 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.122625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-tmp\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.125091 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.125065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfb42f63-665c-4bd7-bf86-233e503c3533-agent-certs\") pod \"konnectivity-agent-mnggj\" (UID: \"cfb42f63-665c-4bd7-bf86-233e503c3533\") " pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:17.125489 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.125457    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:17.125489 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.125485    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:17.125684 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.125500    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ngnk5 for pod openshift-network-diagnostics/network-check-target-7qj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:17.125684 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.125574    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5 podName:cc86fd8d-a0d6-44a0-b8ab-610aa64edcca nodeName:}" failed. No retries permitted until 2026-09-09 21:33:17.625556761 +0000 UTC m=+3.111439785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ngnk5" (UniqueName: "kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5") pod "network-check-target-7qj82" (UID: "cc86fd8d-a0d6-44a0-b8ab-610aa64edcca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:17.127826 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.127750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qnj\" (UniqueName: \"kubernetes.io/projected/0555c16f-7957-4472-a0bf-b9514e19ada1-kube-api-access-v9qnj\") pod \"multus-additional-cni-plugins-hx62l\" (UID: \"0555c16f-7957-4472-a0bf-b9514e19ada1\") " pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.128560 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.128224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgv7j\" (UniqueName: \"kubernetes.io/projected/8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee-kube-api-access-rgv7j\") pod \"aws-ebs-csi-driver-node-65f4q\" (UID: \"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.129608 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.129589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfjd\" (UniqueName: \"kubernetes.io/projected/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-kube-api-access-zzfjd\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:17.130379 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.130348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7d2\" (UniqueName: \"kubernetes.io/projected/8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2-kube-api-access-hz7d2\") pod \"tuned-k44jm\" (UID: \"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2\") " pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.130831 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.130808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2snv\" (UniqueName: \"kubernetes.io/projected/8ed67432-0abd-4885-b0a8-7107e0fe1eb1-kube-api-access-s2snv\") pod \"ovnkube-node-gs2lz\" (UID: \"8ed67432-0abd-4885-b0a8-7107e0fe1eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.132257 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.132234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8gmg\" (UniqueName: \"kubernetes.io/projected/3548b84d-7e5a-43fe-a402-7aa1a566e6d1-kube-api-access-k8gmg\") pod \"iptables-alerter-kvs2d\" (UID: \"3548b84d-7e5a-43fe-a402-7aa1a566e6d1\") " pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.213139 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.213102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c56fx"
Sep 09 21:33:17.222861 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.222832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lrtz6"
Sep 09 21:33:17.234399 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.233773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-klgv6"
Sep 09 21:33:17.243044 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.243013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hx62l"
Sep 09 21:33:17.253008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.252981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kvs2d"
Sep 09 21:33:17.261232 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.261202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:17.270320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.270290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:17.281669 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.281411    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q"
Sep 09 21:33:17.286626 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.286599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k44jm"
Sep 09 21:33:17.624660 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.624618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:17.624897 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.624693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:17.624897 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.624812    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:17.624897 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.624816    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:17.624897 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.624885    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret podName:f85f5db0-2786-426b-a8dc-b6f7d6f37fed nodeName:}" failed. No retries permitted until 2026-09-09 21:33:18.624861348 +0000 UTC m=+4.110744398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret") pod "global-pull-secret-syncer-7dlnc" (UID: "f85f5db0-2786-426b-a8dc-b6f7d6f37fed") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:17.625098 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.624906    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs podName:0ae5a6d6-2f5b-4af6-ab7a-b629addbf014 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:18.624895251 +0000 UTC m=+4.110778267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs") pod "network-metrics-daemon-h2p64" (UID: "0ae5a6d6-2f5b-4af6-ab7a-b629addbf014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:17.725815 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.725768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:17.726019 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.725967    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:17.726019 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.725996    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:17.726019 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.726011    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ngnk5 for pod openshift-network-diagnostics/network-check-target-7qj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:17.726177 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:17.726074    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5 podName:cc86fd8d-a0d6-44a0-b8ab-610aa64edcca nodeName:}" failed. No retries permitted until 2026-09-09 21:33:18.726060395 +0000 UTC m=+4.211943427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngnk5" (UniqueName: "kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5") pod "network-check-target-7qj82" (UID: "cc86fd8d-a0d6-44a0-b8ab-610aa64edcca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:17.948764 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.948710    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 21:28:17 +0000 UTC" deadline="2028-03-23 12:07:50.159533278 +0000 UTC"
Sep 09 21:33:17.948764 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:17.948760    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13454h34m32.210777224s"
Sep 09 21:33:18.039775 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:18.039739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:18.039973 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.039863    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:18.330821 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:18.330570    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9fc8d2_e03f_44e4_89c2_2e6e697f8970.slice/crio-2ecdfd063b6552630922b9de2b3f7fbf2c539f27346d41498c4de06c9263884d WatchSource:0}: Error finding container 2ecdfd063b6552630922b9de2b3f7fbf2c539f27346d41498c4de06c9263884d: Status 404 returned error can't find the container with id 2ecdfd063b6552630922b9de2b3f7fbf2c539f27346d41498c4de06c9263884d
Sep 09 21:33:18.332813 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:18.332714    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb42f63_665c_4bd7_bf86_233e503c3533.slice/crio-03d358bf341ac8e7e7c25fd15467dbb7d2a9c41f226ce73c4eb47179302a501b WatchSource:0}: Error finding container 03d358bf341ac8e7e7c25fd15467dbb7d2a9c41f226ce73c4eb47179302a501b: Status 404 returned error can't find the container with id 03d358bf341ac8e7e7c25fd15467dbb7d2a9c41f226ce73c4eb47179302a501b
Sep 09 21:33:18.335955 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:18.335827    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddb6c2d_e8e5_40a4_b0dd_74ded66c25f6.slice/crio-dd2d131c6fc7b8c86d8b1db08c2c2682e843fac0cbdeb062f2c963484e041e1b WatchSource:0}: Error finding container dd2d131c6fc7b8c86d8b1db08c2c2682e843fac0cbdeb062f2c963484e041e1b: Status 404 returned error can't find the container with id dd2d131c6fc7b8c86d8b1db08c2c2682e843fac0cbdeb062f2c963484e041e1b
Sep 09 21:33:18.337326 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:18.337306    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c59f67_3ceb_4cf5_a041_22852d768ac3.slice/crio-ec0f1fe7738b7dfdf91ec8b6fe912c3eb8e004d4c78f2ba25dfab08864132f14 WatchSource:0}: Error finding container ec0f1fe7738b7dfdf91ec8b6fe912c3eb8e004d4c78f2ba25dfab08864132f14: Status 404 returned error can't find the container with id ec0f1fe7738b7dfdf91ec8b6fe912c3eb8e004d4c78f2ba25dfab08864132f14
Sep 09 21:33:18.633120 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:18.632936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:18.633242 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:18.633143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:18.633242 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.633091    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:18.633340 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.633245    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret podName:f85f5db0-2786-426b-a8dc-b6f7d6f37fed nodeName:}" failed. No retries permitted until 2026-09-09 21:33:20.633227283 +0000 UTC m=+6.119110316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret") pod "global-pull-secret-syncer-7dlnc" (UID: "f85f5db0-2786-426b-a8dc-b6f7d6f37fed") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:18.633340 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.633249    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:18.633340 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.633288    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs podName:0ae5a6d6-2f5b-4af6-ab7a-b629addbf014 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:20.633276041 +0000 UTC m=+6.119159072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs") pod "network-metrics-daemon-h2p64" (UID: "0ae5a6d6-2f5b-4af6-ab7a-b629addbf014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:18.734563 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:18.734524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:18.734789 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.734721    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:18.734789 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.734753    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:18.734789 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.734784    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ngnk5 for pod openshift-network-diagnostics/network-check-target-7qj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:18.734962 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:18.734862    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5 podName:cc86fd8d-a0d6-44a0-b8ab-610aa64edcca nodeName:}" failed. No retries permitted until 2026-09-09 21:33:20.734840081 +0000 UTC m=+6.220723120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngnk5" (UniqueName: "kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5") pod "network-check-target-7qj82" (UID: "cc86fd8d-a0d6-44a0-b8ab-610aa64edcca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:18.829034 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:18.828988    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:33:18.949086 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:18.948900    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 21:28:17 +0000 UTC" deadline="2028-04-16 17:45:14.299233351 +0000 UTC"
Sep 09 21:33:18.949086 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:18.948970    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14036h11m55.350267713s"
Sep 09 21:33:19.040987 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.040910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:19.040987 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.040933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:19.041220 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:19.041052    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:19.041481 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:19.041436    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:19.064207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.064159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q" event={"ID":"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee","Type":"ContainerStarted","Data":"f4fd0068a300312000b971fba50bc9c054b6779a59f1cd4d268dec6687a196b6"}
Sep 09 21:33:19.073262 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.073221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kvs2d" event={"ID":"3548b84d-7e5a-43fe-a402-7aa1a566e6d1","Type":"ContainerStarted","Data":"04392ef1d2cc47940dc965b45ebfaea34012581fb20f006c0c40f7ad32a04129"}
Sep 09 21:33:19.077981 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.077949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lrtz6" event={"ID":"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6","Type":"ContainerStarted","Data":"dd2d131c6fc7b8c86d8b1db08c2c2682e843fac0cbdeb062f2c963484e041e1b"}
Sep 09 21:33:19.087089 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.087054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mnggj" event={"ID":"cfb42f63-665c-4bd7-bf86-233e503c3533","Type":"ContainerStarted","Data":"03d358bf341ac8e7e7c25fd15467dbb7d2a9c41f226ce73c4eb47179302a501b"}
Sep 09 21:33:19.090201 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.090114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-klgv6" event={"ID":"f3c59f67-3ceb-4cf5-a041-22852d768ac3","Type":"ContainerStarted","Data":"ec0f1fe7738b7dfdf91ec8b6fe912c3eb8e004d4c78f2ba25dfab08864132f14"}
Sep 09 21:33:19.097518 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.097393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"03f478913e667d8316e9f8cf82e45a958e26c92a60abb07c6975fe3959ac5a89"}
Sep 09 21:33:19.105038 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.104928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c56fx" event={"ID":"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970","Type":"ContainerStarted","Data":"2ecdfd063b6552630922b9de2b3f7fbf2c539f27346d41498c4de06c9263884d"}
Sep 09 21:33:19.118118 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.118027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal" event={"ID":"bd5a5b93ecd8465e35800ce095df071a","Type":"ContainerStarted","Data":"e950b01634537e7ff2d7355560c543e7bf4fc3e68bc7a159705f8b9b718d5a2f"}
Sep 09 21:33:19.129288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.129252    2563 generic.go:356] "Generic (PLEG): container finished" podID="94d8dd3914b89c4faa93d8b0e772fd80" containerID="e590c79c99d227574f409fd5612634cbc0df8700da39168b65937aa6813251ca" exitCode=0
Sep 09 21:33:19.129464 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.129368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal" event={"ID":"94d8dd3914b89c4faa93d8b0e772fd80","Type":"ContainerDied","Data":"e590c79c99d227574f409fd5612634cbc0df8700da39168b65937aa6813251ca"}
Sep 09 21:33:19.140441 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.140378    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal" podStartSLOduration=3.1403563 podStartE2EDuration="3.1403563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:19.139418844 +0000 UTC m=+4.625301885" watchObservedRunningTime="2026-09-09 21:33:19.1403563 +0000 UTC m=+4.626239343"
Sep 09 21:33:19.144283 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.144248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k44jm" event={"ID":"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2","Type":"ContainerStarted","Data":"32c988527f0c6f1a72fc0296aaa43c3563aa4272db96fe4a921ea6636d57eb28"}
Sep 09 21:33:19.157848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:19.157775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hx62l" event={"ID":"0555c16f-7957-4472-a0bf-b9514e19ada1","Type":"ContainerStarted","Data":"fd28545dd9f4d3e0572f9d51d8ec1442f58c688b7004382387d0fa9147c8fa1e"}
Sep 09 21:33:20.039938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:20.039905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:20.040411 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.040040    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:20.181968 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:20.181329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal" event={"ID":"94d8dd3914b89c4faa93d8b0e772fd80","Type":"ContainerStarted","Data":"8fa1992329a7c3d94b851ae4ed084f5af938f3a3439d200fa7561385c0fe0b27"}
Sep 09 21:33:20.196890 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:20.196434    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal" podStartSLOduration=4.196411606 podStartE2EDuration="4.196411606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:20.195416029 +0000 UTC m=+5.681299067" watchObservedRunningTime="2026-09-09 21:33:20.196411606 +0000 UTC m=+5.682294645"
Sep 09 21:33:20.651984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:20.651940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:20.652164 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:20.652012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:20.652244 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.652171    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:20.652244 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.652237    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs podName:0ae5a6d6-2f5b-4af6-ab7a-b629addbf014 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:24.65221855 +0000 UTC m=+10.138101585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs") pod "network-metrics-daemon-h2p64" (UID: "0ae5a6d6-2f5b-4af6-ab7a-b629addbf014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:20.652675 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.652650    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:20.652807 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.652731    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret podName:f85f5db0-2786-426b-a8dc-b6f7d6f37fed nodeName:}" failed. No retries permitted until 2026-09-09 21:33:24.652698143 +0000 UTC m=+10.138581162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret") pod "global-pull-secret-syncer-7dlnc" (UID: "f85f5db0-2786-426b-a8dc-b6f7d6f37fed") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:20.753051 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:20.752961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:20.753219 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.753147    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:20.753219 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.753169    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:20.753219 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.753181    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ngnk5 for pod openshift-network-diagnostics/network-check-target-7qj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:20.753549 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:20.753238    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5 podName:cc86fd8d-a0d6-44a0-b8ab-610aa64edcca nodeName:}" failed. No retries permitted until 2026-09-09 21:33:24.753221015 +0000 UTC m=+10.239104054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngnk5" (UniqueName: "kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5") pod "network-check-target-7qj82" (UID: "cc86fd8d-a0d6-44a0-b8ab-610aa64edcca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:21.040600 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:21.040044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:21.040600 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:21.040044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:21.040600 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:21.040205    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:21.040600 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:21.040344    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:22.040265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:22.040230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:22.040491 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:22.040416    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:23.040072 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:23.040036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:23.040544 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:23.040175    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:23.040802 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:23.040627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:23.040802 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:23.040759    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:24.040278 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:24.040208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:24.040679 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.040357    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:24.688156 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:24.687293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:24.688156 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:24.687398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:24.688156 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.687524    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:24.688156 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.687594    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret podName:f85f5db0-2786-426b-a8dc-b6f7d6f37fed nodeName:}" failed. No retries permitted until 2026-09-09 21:33:32.68757424 +0000 UTC m=+18.173457266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret") pod "global-pull-secret-syncer-7dlnc" (UID: "f85f5db0-2786-426b-a8dc-b6f7d6f37fed") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:24.688156 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.688070    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:24.688156 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.688122    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs podName:0ae5a6d6-2f5b-4af6-ab7a-b629addbf014 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:32.688109344 +0000 UTC m=+18.173992360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs") pod "network-metrics-daemon-h2p64" (UID: "0ae5a6d6-2f5b-4af6-ab7a-b629addbf014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:24.788337 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:24.788296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:24.788502 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.788445    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:24.788502 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.788462    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:24.788502 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.788471    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ngnk5 for pod openshift-network-diagnostics/network-check-target-7qj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:24.788657 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:24.788513    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5 podName:cc86fd8d-a0d6-44a0-b8ab-610aa64edcca nodeName:}" failed. No retries permitted until 2026-09-09 21:33:32.788500852 +0000 UTC m=+18.274383867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngnk5" (UniqueName: "kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5") pod "network-check-target-7qj82" (UID: "cc86fd8d-a0d6-44a0-b8ab-610aa64edcca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:25.040143 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:25.040066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:25.040143 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:25.040109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:25.040604 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:25.040173    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:25.040604 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:25.040255    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:26.039890 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:26.039821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:26.040181 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:26.039974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:27.040325 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:27.040288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:27.040929 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:27.040333    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:27.040929 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:27.040445    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:27.040929 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:27.040561    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:28.040729 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:28.040676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:28.041171 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:28.040825    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:29.040302 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:29.040267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:29.040463 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:29.040278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:29.040463 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:29.040374    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:29.040538 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:29.040466    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:30.040208 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:30.040166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:30.040661 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:30.040308    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:31.040371 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:31.040332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:31.040821 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:31.040502    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:31.040821 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:31.040546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:31.040821 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:31.040684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:32.040134 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:32.040096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:32.040328 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.040228    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:32.749276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:32.749148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:32.749276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:32.749200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:32.749888 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.749348    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:32.749888 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.749378    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:32.749888 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.749427    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret podName:f85f5db0-2786-426b-a8dc-b6f7d6f37fed nodeName:}" failed. No retries permitted until 2026-09-09 21:33:48.749405179 +0000 UTC m=+34.235288199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret") pod "global-pull-secret-syncer-7dlnc" (UID: "f85f5db0-2786-426b-a8dc-b6f7d6f37fed") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:32.749888 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.749442    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs podName:0ae5a6d6-2f5b-4af6-ab7a-b629addbf014 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:48.749435934 +0000 UTC m=+34.235318950 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs") pod "network-metrics-daemon-h2p64" (UID: "0ae5a6d6-2f5b-4af6-ab7a-b629addbf014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:32.850267 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:32.850229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:32.850451 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.850363    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:32.850451 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.850380    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:32.850451 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.850390    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ngnk5 for pod openshift-network-diagnostics/network-check-target-7qj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:32.850451 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:32.850442    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5 podName:cc86fd8d-a0d6-44a0-b8ab-610aa64edcca nodeName:}" failed. No retries permitted until 2026-09-09 21:33:48.85042929 +0000 UTC m=+34.336312307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngnk5" (UniqueName: "kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5") pod "network-check-target-7qj82" (UID: "cc86fd8d-a0d6-44a0-b8ab-610aa64edcca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:33.040693 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:33.040613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:33.040693 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:33.040643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:33.040936 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:33.040769    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:33.040936 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:33.040868    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:34.040778 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:34.040730    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:34.041182 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:34.040876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:35.040708 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:35.040668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:35.040912 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:35.040785    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:35.041331 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:35.041155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:35.041331 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:35.041260    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:36.040050 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:36.040011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:36.040234 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:36.040151    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:37.040997 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.040626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:37.041886 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.040629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:37.041886 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:37.041099    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:37.041886 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:37.041194    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:37.224283 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.224193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k44jm" event={"ID":"8e9f9c8a-9be3-48e6-8182-f0a0f933fcc2","Type":"ContainerStarted","Data":"690c01628ae433bc99bc02bf140c1a1f86bc87835b8154cb30063af2d75346a6"}
Sep 09 21:33:37.225964 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.225934    2563 generic.go:356] "Generic (PLEG): container finished" podID="0555c16f-7957-4472-a0bf-b9514e19ada1" containerID="10b63362005458c74c5b6c4339205b472509f18bcfdea3a89b277bae3f83349c" exitCode=0
Sep 09 21:33:37.226092 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.226015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hx62l" event={"ID":"0555c16f-7957-4472-a0bf-b9514e19ada1","Type":"ContainerDied","Data":"10b63362005458c74c5b6c4339205b472509f18bcfdea3a89b277bae3f83349c"}
Sep 09 21:33:37.227906 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.227873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q" event={"ID":"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee","Type":"ContainerStarted","Data":"d900faae6aa8e665cfe070b960c127c31e63601472f52b69dc56ca6dd485c5f0"}
Sep 09 21:33:37.229755 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.229704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lrtz6" event={"ID":"fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6","Type":"ContainerStarted","Data":"81216bca3546c0cdb7cc5dc6053b4c7cc4929307c5123b00db49dce26bf856c4"}
Sep 09 21:33:37.231397 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.231371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mnggj" event={"ID":"cfb42f63-665c-4bd7-bf86-233e503c3533","Type":"ContainerStarted","Data":"18d569926fd2e2ff2cd4ad58cc861af91be0bff6afc253e4ed6017733db4c990"}
Sep 09 21:33:37.233133 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.233110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-klgv6" event={"ID":"f3c59f67-3ceb-4cf5-a041-22852d768ac3","Type":"ContainerStarted","Data":"79470ce243905e188ffc2825471debf272028aced605e38d049022073335ea49"}
Sep 09 21:33:37.236530 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.236509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"feb3c56c6c3989a3c748f6c8b1f2ff93679311810a678f614d669f47fc75661c"}
Sep 09 21:33:37.236624 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.236573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"8f16bda91763e4efe1741d224638163688de6742f75c629e1c9476e0d06d8343"}
Sep 09 21:33:37.236624 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.236615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"cf9f2db3c70ba01e82beed4061a5d5ffad2c949b81528669221b36b0bbc6474e"}
Sep 09 21:33:37.236734 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.236663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"8f9a3ce76e0e7e6f7c1689fef8d8b5e872154ea338cd70ad25381ae725606266"}
Sep 09 21:33:37.236734 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.236700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"9857f42af7533c04d23355b62d5ad433babc667e4b7d62274c76a88ba2f4533c"}
Sep 09 21:33:37.236811 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.236754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"72fa4cd2f17d9e0bdfce355fe31264f9649e86ce034530ecf96395544653450f"}
Sep 09 21:33:37.238111 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.238088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c56fx" event={"ID":"cc9fc8d2-e03f-44e4-89c2-2e6e697f8970","Type":"ContainerStarted","Data":"d777bbb2c34df95b472c756f4681a98f48aebb3367656ce5c71ad07800373cf6"}
Sep 09 21:33:37.240130 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.240090    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k44jm" podStartSLOduration=4.337038422 podStartE2EDuration="22.240078917s" totalImagesPullingTime="17.903040495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:37.23983319 +0000 UTC m=+22.725716232" watchObservedRunningTime="2026-09-09 21:33:37.240078917 +0000 UTC m=+22.725961981"
Sep 09 21:33:37.257048 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.257008    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c56fx" podStartSLOduration=4.274722679 podStartE2EDuration="22.25699445s" totalImagesPullingTime="17.982271771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:37.256436873 +0000 UTC m=+22.742319912" watchObservedRunningTime="2026-09-09 21:33:37.25699445 +0000 UTC m=+22.742877488"
Sep 09 21:33:37.291625 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.291579    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mnggj" podStartSLOduration=8.404906076 podStartE2EDuration="22.291563392s" totalImagesPullingTime="13.886657316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:37.291142676 +0000 UTC m=+22.777025726" watchObservedRunningTime="2026-09-09 21:33:37.291563392 +0000 UTC m=+22.777446421"
Sep 09 21:33:37.305133 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.305083    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lrtz6" podStartSLOduration=4.42322409 podStartE2EDuration="22.305065121s" totalImagesPullingTime="17.881841031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:37.304972819 +0000 UTC m=+22.790855860" watchObservedRunningTime="2026-09-09 21:33:37.305065121 +0000 UTC m=+22.790948162"
Sep 09 21:33:37.319230 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.319185    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-klgv6" podStartSLOduration=4.438825876 podStartE2EDuration="22.319171246s" totalImagesPullingTime="17.88034537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:37.319114663 +0000 UTC m=+22.804997702" watchObservedRunningTime="2026-09-09 21:33:37.319171246 +0000 UTC m=+22.805054308"
Sep 09 21:33:37.751508 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.751480    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 21:33:37.978175 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.978030    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T21:33:37.751503939Z","UUID":"a78d5f13-b3ed-4fdd-9e29-5205905c64fb","Handler":null,"Name":"","Endpoint":""}
Sep 09 21:33:37.979528 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.979506    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 21:33:37.979665 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:37.979540    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 21:33:38.040251 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:38.040217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:38.040440 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:38.040355    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:38.242325 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:38.242235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q" event={"ID":"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee","Type":"ContainerStarted","Data":"9c421de01f7972ac9c5996a2773d357cf99e99cae90808aec35f5e716c6cffcf"}
Sep 09 21:33:38.244143 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:38.244086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kvs2d" event={"ID":"3548b84d-7e5a-43fe-a402-7aa1a566e6d1","Type":"ContainerStarted","Data":"3806f20d75ffdf96a23c344396a27bc76c897a08abbc81edc36da8ea1a4e123f"}
Sep 09 21:33:38.252147 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:38.252126    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:38.252756 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:38.252738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:38.258587 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:38.258542    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kvs2d" podStartSLOduration=5.383843878 podStartE2EDuration="23.258528586s" totalImagesPullingTime="17.874684708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:38.258234217 +0000 UTC m=+23.744117255" watchObservedRunningTime="2026-09-09 21:33:38.258528586 +0000 UTC m=+23.744411623"
Sep 09 21:33:39.040065 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:39.040037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:39.040188 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:39.040167    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:39.040288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:39.040264    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:39.040420 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:39.040399    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:39.250797 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:39.250704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"7c6f79fd042750ad6c70f1f1426e74dbe090320b58d10bb9ebaed035aeca8cc1"}
Sep 09 21:33:39.251208 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:39.251064    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:39.251461 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:39.251438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mnggj"
Sep 09 21:33:40.039782 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:40.039752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:40.039967 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:40.039882    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:40.255880 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:40.255303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q" event={"ID":"8a4e42b0-5bb5-4e3e-b3f7-7a68baf9c9ee","Type":"ContainerStarted","Data":"f903d98c3008830ea28399e44fea9ae365fcdf7c2593bbc1468a1bc238729da3"}
Sep 09 21:33:40.272813 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:40.272757    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65f4q" podStartSLOduration=4.681484724 podStartE2EDuration="25.272739651s" totalImagesPullingTime="20.591254927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:40.27226113 +0000 UTC m=+25.758144168" watchObservedRunningTime="2026-09-09 21:33:40.272739651 +0000 UTC m=+25.758622690"
Sep 09 21:33:41.039785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:41.039753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:41.039785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:41.039787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:41.040032 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:41.039876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:41.040032 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:41.039999    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:42.040479 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:42.040432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:42.040969 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:42.040571    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:43.040396 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:43.040151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:43.040531 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:43.040191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:43.040531 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:43.040453    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:43.040531 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:43.040504    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:43.265080 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:43.265041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" event={"ID":"8ed67432-0abd-4885-b0a8-7107e0fe1eb1","Type":"ContainerStarted","Data":"46cb2f9f7f92f49269a40890006e5d5340e361687eb17348ca090989ed223c31"}
Sep 09 21:33:43.265321 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:43.265299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:43.266984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:43.266960    2563 generic.go:356] "Generic (PLEG): container finished" podID="0555c16f-7957-4472-a0bf-b9514e19ada1" containerID="c7eadb1620948f4cd4f0df11de5b18fdd2daab7b4eda12499a6374609a4d6efa" exitCode=0
Sep 09 21:33:43.267119 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:43.267000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hx62l" event={"ID":"0555c16f-7957-4472-a0bf-b9514e19ada1","Type":"ContainerDied","Data":"c7eadb1620948f4cd4f0df11de5b18fdd2daab7b4eda12499a6374609a4d6efa"}
Sep 09 21:33:43.281796 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:43.281761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:43.292164 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:43.292061    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz" podStartSLOduration=10.352772463 podStartE2EDuration="28.292044046s" totalImagesPullingTime="17.939271583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:43.290133499 +0000 UTC m=+28.776016537" watchObservedRunningTime="2026-09-09 21:33:43.292044046 +0000 UTC m=+28.777927085"
Sep 09 21:33:44.040023 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.039974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:44.040313 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:44.040130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:44.231762 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.231699    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:44.270161 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.270128    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:44.286406 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.286379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:33:44.318586 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.318500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7dlnc"]
Sep 09 21:33:44.318764 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.318654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:44.318842 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:44.318775    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:44.319949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.319927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h2p64"]
Sep 09 21:33:44.320106 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.320025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:44.320173 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:44.320128    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:44.330396 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.330339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7qj82"]
Sep 09 21:33:44.330508 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:44.330429    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:44.330508 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:44.330493    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:45.273893 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:45.273861    2563 generic.go:356] "Generic (PLEG): container finished" podID="0555c16f-7957-4472-a0bf-b9514e19ada1" containerID="e858eff84504d42d10d250dcc565e671f6e64b3171dc976ff0343517a93610fd" exitCode=0
Sep 09 21:33:45.274358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:45.273943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hx62l" event={"ID":"0555c16f-7957-4472-a0bf-b9514e19ada1","Type":"ContainerDied","Data":"e858eff84504d42d10d250dcc565e671f6e64b3171dc976ff0343517a93610fd"}
Sep 09 21:33:46.040333 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:46.040296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:46.040333 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:46.040330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:46.040636 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:46.040302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:46.040636 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:46.040432    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:46.040636 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:46.040474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:46.040636 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:46.040559    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:47.281122 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:47.280918    2563 generic.go:356] "Generic (PLEG): container finished" podID="0555c16f-7957-4472-a0bf-b9514e19ada1" containerID="908547f44cc249ff33e431bb361efe51a4f4b1c90c788b79fc82bf7cbe4eafd9" exitCode=0
Sep 09 21:33:47.281122 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:47.281002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hx62l" event={"ID":"0555c16f-7957-4472-a0bf-b9514e19ada1","Type":"ContainerDied","Data":"908547f44cc249ff33e431bb361efe51a4f4b1c90c788b79fc82bf7cbe4eafd9"}
Sep 09 21:33:48.040405 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:48.040365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:48.040405 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:48.040387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:48.040405 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:48.040402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:48.040655 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.040498    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2p64" podUID="0ae5a6d6-2f5b-4af6-ab7a-b629addbf014"
Sep 09 21:33:48.040655 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.040539    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qj82" podUID="cc86fd8d-a0d6-44a0-b8ab-610aa64edcca"
Sep 09 21:33:48.040655 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.040620    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7dlnc" podUID="f85f5db0-2786-426b-a8dc-b6f7d6f37fed"
Sep 09 21:33:48.767343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:48.767298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:48.767343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:48.767354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:48.767835 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.767459    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:48.767835 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.767459    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:48.767835 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.767531    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs podName:0ae5a6d6-2f5b-4af6-ab7a-b629addbf014 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:20.767508189 +0000 UTC m=+66.253391225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs") pod "network-metrics-daemon-h2p64" (UID: "0ae5a6d6-2f5b-4af6-ab7a-b629addbf014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:33:48.767835 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.767545    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret podName:f85f5db0-2786-426b-a8dc-b6f7d6f37fed nodeName:}" failed. No retries permitted until 2026-09-09 21:34:20.767539463 +0000 UTC m=+66.253422493 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret") pod "global-pull-secret-syncer-7dlnc" (UID: "f85f5db0-2786-426b-a8dc-b6f7d6f37fed") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:33:48.868297 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:48.868256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:48.868487 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.868413    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:33:48.868487 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.868443    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:33:48.868487 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.868459    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ngnk5 for pod openshift-network-diagnostics/network-check-target-7qj82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:48.868647 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:48.868517    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5 podName:cc86fd8d-a0d6-44a0-b8ab-610aa64edcca nodeName:}" failed. No retries permitted until 2026-09-09 21:34:20.868498805 +0000 UTC m=+66.354381834 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngnk5" (UniqueName: "kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5") pod "network-check-target-7qj82" (UID: "cc86fd8d-a0d6-44a0-b8ab-610aa64edcca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:33:49.302563 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.302530    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeReady"
Sep 09 21:33:49.302768 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.302699    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 21:33:49.345607 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.345574    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cplkq"]
Sep 09 21:33:49.361301 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.359365    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cx5mp"]
Sep 09 21:33:49.361301 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.359500    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.362547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.362480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4v4f\""
Sep 09 21:33:49.362671 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.362640    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 21:33:49.362773 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.362754    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 21:33:49.372611 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.372585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cplkq"]
Sep 09 21:33:49.372749 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.372732    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cx5mp"
Sep 09 21:33:49.372803 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.372762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cx5mp"]
Sep 09 21:33:49.376989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.376753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bhsnj\""
Sep 09 21:33:49.376989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.376800    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 21:33:49.376989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.376899    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 21:33:49.376989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.376890    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 21:33:49.473745 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.473630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.473745 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.473704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-config-volume\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.473981 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.473750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnbl\" (UniqueName: \"kubernetes.io/projected/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-kube-api-access-bvnbl\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.473981 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.473771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44420c86-87ae-49ea-9aa3-7c62ed0caf2f-cert\") pod \"ingress-canary-cx5mp\" (UID: \"44420c86-87ae-49ea-9aa3-7c62ed0caf2f\") " pod="openshift-ingress-canary/ingress-canary-cx5mp"
Sep 09 21:33:49.473981 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.473788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhr5\" (UniqueName: \"kubernetes.io/projected/44420c86-87ae-49ea-9aa3-7c62ed0caf2f-kube-api-access-wzhr5\") pod \"ingress-canary-cx5mp\" (UID: \"44420c86-87ae-49ea-9aa3-7c62ed0caf2f\") " pod="openshift-ingress-canary/ingress-canary-cx5mp"
Sep 09 21:33:49.473981 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.473814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-tmp-dir\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.574912 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.574865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-config-volume\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.574912 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.574911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnbl\" (UniqueName: \"kubernetes.io/projected/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-kube-api-access-bvnbl\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.575218 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.574928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44420c86-87ae-49ea-9aa3-7c62ed0caf2f-cert\") pod \"ingress-canary-cx5mp\" (UID: \"44420c86-87ae-49ea-9aa3-7c62ed0caf2f\") " pod="openshift-ingress-canary/ingress-canary-cx5mp"
Sep 09 21:33:49.575218 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.574944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhr5\" (UniqueName: \"kubernetes.io/projected/44420c86-87ae-49ea-9aa3-7c62ed0caf2f-kube-api-access-wzhr5\") pod \"ingress-canary-cx5mp\" (UID: \"44420c86-87ae-49ea-9aa3-7c62ed0caf2f\") " pod="openshift-ingress-canary/ingress-canary-cx5mp"
Sep 09 21:33:49.575218 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.574965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-tmp-dir\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.575218 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.575021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.575218 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:49.575145    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:49.575419 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:49.575225    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls podName:c70d3852-c3a8-4a65-b4bc-661f6a30a9d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:50.075205099 +0000 UTC m=+35.561088129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls") pod "dns-default-cplkq" (UID: "c70d3852-c3a8-4a65-b4bc-661f6a30a9d4") : secret "dns-default-metrics-tls" not found
Sep 09 21:33:49.575419 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.575359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-tmp-dir\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.575525 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.575443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-config-volume\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.579290 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.579264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44420c86-87ae-49ea-9aa3-7c62ed0caf2f-cert\") pod \"ingress-canary-cx5mp\" (UID: \"44420c86-87ae-49ea-9aa3-7c62ed0caf2f\") " pod="openshift-ingress-canary/ingress-canary-cx5mp"
Sep 09 21:33:49.585420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.585398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnbl\" (UniqueName: \"kubernetes.io/projected/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-kube-api-access-bvnbl\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:49.585638 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.585616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhr5\" (UniqueName: \"kubernetes.io/projected/44420c86-87ae-49ea-9aa3-7c62ed0caf2f-kube-api-access-wzhr5\") pod \"ingress-canary-cx5mp\" (UID: \"44420c86-87ae-49ea-9aa3-7c62ed0caf2f\") " pod="openshift-ingress-canary/ingress-canary-cx5mp"
Sep 09 21:33:49.685369 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:49.685333    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cx5mp"
Sep 09 21:33:50.004168 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.004133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cx5mp"]
Sep 09 21:33:50.008347 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:33:50.008320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44420c86_87ae_49ea_9aa3_7c62ed0caf2f.slice/crio-55c7618b1f162e170101da0ebccfb5b90b654a7aca42092a0297ef17a2572b40 WatchSource:0}: Error finding container 55c7618b1f162e170101da0ebccfb5b90b654a7aca42092a0297ef17a2572b40: Status 404 returned error can't find the container with id 55c7618b1f162e170101da0ebccfb5b90b654a7aca42092a0297ef17a2572b40
Sep 09 21:33:50.040603 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.040528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:33:50.040776 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.040533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:33:50.040776 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.040660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:33:50.043700 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.043682    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 21:33:50.043700 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.043702    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 21:33:50.044873 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.044849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 21:33:50.045008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.044876    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 21:33:50.045008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.044895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kms6v\""
Sep 09 21:33:50.045008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.044878    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5t55x\""
Sep 09 21:33:50.080395 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.080355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:50.080562 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:50.080542    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:50.080629 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:50.080612    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls podName:c70d3852-c3a8-4a65-b4bc-661f6a30a9d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:51.080591409 +0000 UTC m=+36.566474443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls") pod "dns-default-cplkq" (UID: "c70d3852-c3a8-4a65-b4bc-661f6a30a9d4") : secret "dns-default-metrics-tls" not found
Sep 09 21:33:50.290747 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:50.290637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cx5mp" event={"ID":"44420c86-87ae-49ea-9aa3-7c62ed0caf2f","Type":"ContainerStarted","Data":"55c7618b1f162e170101da0ebccfb5b90b654a7aca42092a0297ef17a2572b40"}
Sep 09 21:33:51.090780 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:51.089934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:51.090780 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:51.090341    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:51.090780 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:51.090404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls podName:c70d3852-c3a8-4a65-b4bc-661f6a30a9d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:53.09038997 +0000 UTC m=+38.576273002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls") pod "dns-default-cplkq" (UID: "c70d3852-c3a8-4a65-b4bc-661f6a30a9d4") : secret "dns-default-metrics-tls" not found
Sep 09 21:33:53.108528 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:53.108492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:53.109057 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:53.108662    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:53.109057 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:53.108764    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls podName:c70d3852-c3a8-4a65-b4bc-661f6a30a9d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:33:57.108742971 +0000 UTC m=+42.594626004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls") pod "dns-default-cplkq" (UID: "c70d3852-c3a8-4a65-b4bc-661f6a30a9d4") : secret "dns-default-metrics-tls" not found
Sep 09 21:33:54.307687 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:54.307586    2563 generic.go:356] "Generic (PLEG): container finished" podID="0555c16f-7957-4472-a0bf-b9514e19ada1" containerID="0d7636416a0755f72e6ededaba671643f6fe9add5740beabd2f16bea9448e9f6" exitCode=0
Sep 09 21:33:54.308213 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:54.307672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hx62l" event={"ID":"0555c16f-7957-4472-a0bf-b9514e19ada1","Type":"ContainerDied","Data":"0d7636416a0755f72e6ededaba671643f6fe9add5740beabd2f16bea9448e9f6"}
Sep 09 21:33:54.309273 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:54.309243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cx5mp" event={"ID":"44420c86-87ae-49ea-9aa3-7c62ed0caf2f","Type":"ContainerStarted","Data":"48b18eec295a847a4be7cb21c16b9118096fb5e146050ba48f6df11219b634dc"}
Sep 09 21:33:54.345368 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:54.345313    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cx5mp" podStartSLOduration=1.626631545 podStartE2EDuration="5.345292466s" totalImagesPullingTime="3.718660921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:54.345065934 +0000 UTC m=+39.830948971" watchObservedRunningTime="2026-09-09 21:33:54.345292466 +0000 UTC m=+39.831175504"
Sep 09 21:33:55.315945 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:55.315026    2563 generic.go:356] "Generic (PLEG): container finished" podID="0555c16f-7957-4472-a0bf-b9514e19ada1" containerID="5d0fdf6d05571f2cf9276f3a259c8d3b869045f551a225855bd704b08c304703" exitCode=0
Sep 09 21:33:55.315945 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:55.315841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hx62l" event={"ID":"0555c16f-7957-4472-a0bf-b9514e19ada1","Type":"ContainerDied","Data":"5d0fdf6d05571f2cf9276f3a259c8d3b869045f551a225855bd704b08c304703"}
Sep 09 21:33:56.320527 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:56.320495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hx62l" event={"ID":"0555c16f-7957-4472-a0bf-b9514e19ada1","Type":"ContainerStarted","Data":"31527938453da34b97dd7344a7abae848fc380495c6516637c37921bfabb8f61"}
Sep 09 21:33:56.341598 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:56.341484    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hx62l" podStartSLOduration=9.492786792 podStartE2EDuration="41.341465376s" totalImagesPullingTime="31.848678584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:56.341327311 +0000 UTC m=+41.827210362" watchObservedRunningTime="2026-09-09 21:33:56.341465376 +0000 UTC m=+41.827348414"
Sep 09 21:33:57.138206 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:33:57.138163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:33:57.138377 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:57.138325    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:33:57.138427 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:33:57.138400    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls podName:c70d3852-c3a8-4a65-b4bc-661f6a30a9d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:05.138384038 +0000 UTC m=+50.624267053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls") pod "dns-default-cplkq" (UID: "c70d3852-c3a8-4a65-b4bc-661f6a30a9d4") : secret "dns-default-metrics-tls" not found
Sep 09 21:34:05.194785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.194738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:34:05.195136 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:05.194907    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:34:05.195136 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:05.194976    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls podName:c70d3852-c3a8-4a65-b4bc-661f6a30a9d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:34:21.194959793 +0000 UTC m=+66.680842831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls") pod "dns-default-cplkq" (UID: "c70d3852-c3a8-4a65-b4bc-661f6a30a9d4") : secret "dns-default-metrics-tls" not found
Sep 09 21:34:05.605609 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.605575    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-9w7vx"]
Sep 09 21:34:05.609262 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.609244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-9w7vx"
Sep 09 21:34:05.612294 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.612270    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"]
Sep 09 21:34:05.612753 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.612731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xv728\""
Sep 09 21:34:05.617350 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.617309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:05.618362 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.618341    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-9w7vx"]
Sep 09 21:34:05.620412 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.620395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 21:34:05.620798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.620778    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:34:05.620897 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.620801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 21:34:05.620897 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.620787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r9ssj\""
Sep 09 21:34:05.629695 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.629666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"]
Sep 09 21:34:05.698314 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.698279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25cw\" (UniqueName: \"kubernetes.io/projected/f3a13cf8-941e-4fc8-8908-30c4cd7c083b-kube-api-access-c25cw\") pod \"network-check-source-685456655f-9w7vx\" (UID: \"f3a13cf8-941e-4fc8-8908-30c4cd7c083b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-9w7vx"
Sep 09 21:34:05.711216 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.711178    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cc4878bb4-sfpmd"]
Sep 09 21:34:05.712957 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.712940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.715671 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.715647    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 21:34:05.715671 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.715653    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 21:34:05.715860 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.715773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 21:34:05.715943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.715930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ts5sm\""
Sep 09 21:34:05.721277 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.721254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 21:34:05.725395 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.725374    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc4878bb4-sfpmd"]
Sep 09 21:34:05.799621 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-image-registry-private-configuration\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.799822 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-trusted-ca\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.799822 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.799822 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94lcb\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-kube-api-access-94lcb\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.799822 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:05.799822 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-bound-sa-token\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.799822 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25cw\" (UniqueName: \"kubernetes.io/projected/f3a13cf8-941e-4fc8-8908-30c4cd7c083b-kube-api-access-c25cw\") pod \"network-check-source-685456655f-9w7vx\" (UID: \"f3a13cf8-941e-4fc8-8908-30c4cd7c083b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-9w7vx"
Sep 09 21:34:05.800119 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b294a07-8857-420a-bfec-80269ffaf9bc-ca-trust-extracted\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.800119 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-certificates\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.800119 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjx4\" (UniqueName: \"kubernetes.io/projected/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-kube-api-access-whjx4\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:05.800119 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.799925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-installation-pull-secrets\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.813452 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.813416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25cw\" (UniqueName: \"kubernetes.io/projected/f3a13cf8-941e-4fc8-8908-30c4cd7c083b-kube-api-access-c25cw\") pod \"network-check-source-685456655f-9w7vx\" (UID: \"f3a13cf8-941e-4fc8-8908-30c4cd7c083b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-9w7vx"
Sep 09 21:34:05.900914 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.900811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-trusted-ca\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.900914 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.900866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.900914 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.900882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94lcb\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-kube-api-access-94lcb\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.901153 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:05.901153 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:05.901032    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:34:05.901153 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-bound-sa-token\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.901153 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:05.901054    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc4878bb4-sfpmd: secret "image-registry-tls" not found
Sep 09 21:34:05.901153 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b294a07-8857-420a-bfec-80269ffaf9bc-ca-trust-extracted\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.901153 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:05.901101    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls podName:3b294a07-8857-420a-bfec-80269ffaf9bc nodeName:}" failed. No retries permitted until 2026-09-09 21:34:06.401087352 +0000 UTC m=+51.886970399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls") pod "image-registry-cc4878bb4-sfpmd" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc") : secret "image-registry-tls" not found
Sep 09 21:34:05.901153 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-certificates\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.901490 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:05.901188    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:34:05.901490 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjx4\" (UniqueName: \"kubernetes.io/projected/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-kube-api-access-whjx4\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:05.901490 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:05.901274    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls podName:7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b nodeName:}" failed. No retries permitted until 2026-09-09 21:34:06.401253968 +0000 UTC m=+51.887137000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8ppsw" (UID: "7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b") : secret "samples-operator-tls" not found
Sep 09 21:34:05.901490 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-installation-pull-secrets\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.901490 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-image-registry-private-configuration\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.901746 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b294a07-8857-420a-bfec-80269ffaf9bc-ca-trust-extracted\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.901819 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.901801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-certificates\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.902307 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.902284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-trusted-ca\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.903902 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.903875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-image-registry-private-configuration\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.903983 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.903909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-installation-pull-secrets\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.910035 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.910011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjx4\" (UniqueName: \"kubernetes.io/projected/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-kube-api-access-whjx4\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:05.910284 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.910264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-bound-sa-token\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.910558 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.910540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94lcb\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-kube-api-access-94lcb\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:05.918492 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:05.918473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-9w7vx"
Sep 09 21:34:06.184338 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:06.184261    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-9w7vx"]
Sep 09 21:34:06.187798 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:06.187773    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a13cf8_941e_4fc8_8908_30c4cd7c083b.slice/crio-4b08691e43e6fb388ec82bec4e5c97b48dcd23a66bdccc129a0803f7aa5fdde6 WatchSource:0}: Error finding container 4b08691e43e6fb388ec82bec4e5c97b48dcd23a66bdccc129a0803f7aa5fdde6: Status 404 returned error can't find the container with id 4b08691e43e6fb388ec82bec4e5c97b48dcd23a66bdccc129a0803f7aa5fdde6
Sep 09 21:34:06.345020 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:06.344979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-9w7vx" event={"ID":"f3a13cf8-941e-4fc8-8908-30c4cd7c083b","Type":"ContainerStarted","Data":"4b08691e43e6fb388ec82bec4e5c97b48dcd23a66bdccc129a0803f7aa5fdde6"}
Sep 09 21:34:06.406639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:06.406599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:06.406639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:06.406639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:06.406862 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:06.406768    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:34:06.406862 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:06.406774    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:34:06.406862 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:06.406792    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc4878bb4-sfpmd: secret "image-registry-tls" not found
Sep 09 21:34:06.406862 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:06.406836    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls podName:7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b nodeName:}" failed. No retries permitted until 2026-09-09 21:34:07.406821481 +0000 UTC m=+52.892704497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8ppsw" (UID: "7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b") : secret "samples-operator-tls" not found
Sep 09 21:34:06.406862 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:06.406848    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls podName:3b294a07-8857-420a-bfec-80269ffaf9bc nodeName:}" failed. No retries permitted until 2026-09-09 21:34:07.406842735 +0000 UTC m=+52.892725750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls") pod "image-registry-cc4878bb4-sfpmd" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc") : secret "image-registry-tls" not found
Sep 09 21:34:07.415440 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:07.415404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:07.415991 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:07.415447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:07.415991 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:07.415574    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:34:07.415991 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:07.415578    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:34:07.415991 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:07.415595    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc4878bb4-sfpmd: secret "image-registry-tls" not found
Sep 09 21:34:07.415991 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:07.415652    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls podName:7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b nodeName:}" failed. No retries permitted until 2026-09-09 21:34:09.415634535 +0000 UTC m=+54.901517567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8ppsw" (UID: "7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b") : secret "samples-operator-tls" not found
Sep 09 21:34:07.415991 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:07.415666    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls podName:3b294a07-8857-420a-bfec-80269ffaf9bc nodeName:}" failed. No retries permitted until 2026-09-09 21:34:09.415660504 +0000 UTC m=+54.901543520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls") pod "image-registry-cc4878bb4-sfpmd" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc") : secret "image-registry-tls" not found
Sep 09 21:34:09.430083 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:09.430050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:09.430083 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:09.430089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:09.430535 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:09.430206    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:34:09.430535 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:09.430235    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:34:09.430535 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:09.430257    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc4878bb4-sfpmd: secret "image-registry-tls" not found
Sep 09 21:34:09.430535 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:09.430261    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls podName:7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b nodeName:}" failed. No retries permitted until 2026-09-09 21:34:13.43024803 +0000 UTC m=+58.916131047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8ppsw" (UID: "7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b") : secret "samples-operator-tls" not found
Sep 09 21:34:09.430535 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:09.430312    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls podName:3b294a07-8857-420a-bfec-80269ffaf9bc nodeName:}" failed. No retries permitted until 2026-09-09 21:34:13.430292944 +0000 UTC m=+58.916175974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls") pod "image-registry-cc4878bb4-sfpmd" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc") : secret "image-registry-tls" not found
Sep 09 21:34:11.358576 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.358530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-9w7vx" event={"ID":"f3a13cf8-941e-4fc8-8908-30c4cd7c083b","Type":"ContainerStarted","Data":"6ac9d9515ef5ca7306095857cd7711868f6c099b5cbcbb6e704f600f0ea0a8c9"}
Sep 09 21:34:11.373456 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.373399    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-9w7vx" podStartSLOduration=2.132566976 podStartE2EDuration="6.373382411s" totalImagesPullingTime="4.240815435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:11.373157299 +0000 UTC m=+56.859040336" watchObservedRunningTime="2026-09-09 21:34:11.373382411 +0000 UTC m=+56.859265450"
Sep 09 21:34:11.606073 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.606032    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxrq"]
Sep 09 21:34:11.608319 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.608296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.611126 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.611053    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 21:34:11.611126 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.611097    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b8s76\""
Sep 09 21:34:11.611280 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.611205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 21:34:11.611357 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.611340    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 21:34:11.612200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.612175    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 21:34:11.623872 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.623845    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxrq"]
Sep 09 21:34:11.748246 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.748206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.748246 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.748248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvdb\" (UniqueName: \"kubernetes.io/projected/9848c116-2d5e-4afd-abad-74193bc624cf-kube-api-access-fbvdb\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.748453 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.748408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9848c116-2d5e-4afd-abad-74193bc624cf-data-volume\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.748513 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.748467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9848c116-2d5e-4afd-abad-74193bc624cf-crio-socket\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.748513 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.748507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.849011 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.848967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9848c116-2d5e-4afd-abad-74193bc624cf-data-volume\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.849162 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.849061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9848c116-2d5e-4afd-abad-74193bc624cf-crio-socket\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.849162 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.849085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.849162 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.849113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.849162 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.849140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvdb\" (UniqueName: \"kubernetes.io/projected/9848c116-2d5e-4afd-abad-74193bc624cf-kube-api-access-fbvdb\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.849308 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.849195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9848c116-2d5e-4afd-abad-74193bc624cf-crio-socket\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.849308 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:11.849262    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:11.849368 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:11.849338    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:12.349321405 +0000 UTC m=+57.835204421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:11.849405 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:11.849263    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:11.849455 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.849361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9848c116-2d5e-4afd-abad-74193bc624cf-data-volume\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:11.849486 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:11.849458    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:12.349444511 +0000 UTC m=+57.835327541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:11.858662 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:11.858642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvdb\" (UniqueName: \"kubernetes.io/projected/9848c116-2d5e-4afd-abad-74193bc624cf-kube-api-access-fbvdb\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:12.353292 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:12.353248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:12.353292 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:12.353300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:12.353498 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:12.353369    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:12.353498 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:12.353398    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:12.353498 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:12.353438    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:13.353419909 +0000 UTC m=+58.839302947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:12.353498 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:12.353464    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:13.353450148 +0000 UTC m=+58.839333184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:13.360979 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:13.360932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:13.361374 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:13.360998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:13.361374 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.361097    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:13.361374 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.361155    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:13.361374 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.361180    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:15.361161804 +0000 UTC m=+60.847044821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:13.361374 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.361211    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:15.361198557 +0000 UTC m=+60.847081602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:13.461440 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:13.461390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:13.461440 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:13.461444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:13.461690 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.461544    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:34:13.461690 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.461553    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:34:13.461690 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.461574    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc4878bb4-sfpmd: secret "image-registry-tls" not found
Sep 09 21:34:13.461690 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.461603    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls podName:7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b nodeName:}" failed. No retries permitted until 2026-09-09 21:34:21.461588783 +0000 UTC m=+66.947471799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8ppsw" (UID: "7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b") : secret "samples-operator-tls" not found
Sep 09 21:34:13.461690 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:13.461624    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls podName:3b294a07-8857-420a-bfec-80269ffaf9bc nodeName:}" failed. No retries permitted until 2026-09-09 21:34:21.461608214 +0000 UTC m=+66.947491245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls") pod "image-registry-cc4878bb4-sfpmd" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc") : secret "image-registry-tls" not found
Sep 09 21:34:14.369004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:14.368974    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lrtz6_fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6/dns-node-resolver/0.log"
Sep 09 21:34:15.369499 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:15.369471    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-klgv6_f3c59f67-3ceb-4cf5-a041-22852d768ac3/node-ca/0.log"
Sep 09 21:34:15.375890 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:15.375866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:15.375964 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:15.375919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:15.376048 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:15.376031    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:15.376103 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:15.376040    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:15.376103 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:15.376100    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:19.376084733 +0000 UTC m=+64.861967775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:15.376180 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:15.376113    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:19.37610713 +0000 UTC m=+64.861990145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:16.169946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:16.169921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cx5mp_44420c86-87ae-49ea-9aa3-7c62ed0caf2f/serve-healthcheck-canary/0.log"
Sep 09 21:34:16.287201 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:16.287176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gs2lz"
Sep 09 21:34:19.409607 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:19.409574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:19.409986 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:19.409620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:19.409986 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:19.409749    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:19.409986 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:19.409758    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:34:19.409986 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:19.409818    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:27.409803438 +0000 UTC m=+72.895686467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : configmap "kube-rbac-proxy" not found
Sep 09 21:34:19.409986 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:19.409831    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls podName:9848c116-2d5e-4afd-abad-74193bc624cf nodeName:}" failed. No retries permitted until 2026-09-09 21:34:27.409825427 +0000 UTC m=+72.895708443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxrq" (UID: "9848c116-2d5e-4afd-abad-74193bc624cf") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:34:20.819043 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.819002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:34:20.819043 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.819053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:34:20.822268 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.822250    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 21:34:20.822320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.822300    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 21:34:20.829540 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:20.829516    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 21:34:20.829602 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:34:20.829591    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs podName:0ae5a6d6-2f5b-4af6-ab7a-b629addbf014 nodeName:}" failed. No retries permitted until 2026-09-09 21:35:24.829567233 +0000 UTC m=+130.315450253 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs") pod "network-metrics-daemon-h2p64" (UID: "0ae5a6d6-2f5b-4af6-ab7a-b629addbf014") : secret "metrics-daemon-secret" not found
Sep 09 21:34:20.831826 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.831796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f85f5db0-2786-426b-a8dc-b6f7d6f37fed-original-pull-secret\") pod \"global-pull-secret-syncer-7dlnc\" (UID: \"f85f5db0-2786-426b-a8dc-b6f7d6f37fed\") " pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:34:20.920166 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.920133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:34:20.922756 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.922737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnk5\" (UniqueName: \"kubernetes.io/projected/cc86fd8d-a0d6-44a0-b8ab-610aa64edcca-kube-api-access-ngnk5\") pod \"network-check-target-7qj82\" (UID: \"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca\") " pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:34:20.951663 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.951619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7dlnc"
Sep 09 21:34:20.968376 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.968350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kms6v\""
Sep 09 21:34:20.976590 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:20.976562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:34:21.222511 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.222477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:34:21.223288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.223266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7dlnc"]
Sep 09 21:34:21.225006 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.224980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70d3852-c3a8-4a65-b4bc-661f6a30a9d4-metrics-tls\") pod \"dns-default-cplkq\" (UID: \"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4\") " pod="openshift-dns/dns-default-cplkq"
Sep 09 21:34:21.225587 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:21.225564    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85f5db0_2786_426b_a8dc_b6f7d6f37fed.slice/crio-bc11dd925cd8950e407613d34c4c542f5e4ab432696a9a45ae0ca19a74e5ecac WatchSource:0}: Error finding container bc11dd925cd8950e407613d34c4c542f5e4ab432696a9a45ae0ca19a74e5ecac: Status 404 returned error can't find the container with id bc11dd925cd8950e407613d34c4c542f5e4ab432696a9a45ae0ca19a74e5ecac
Sep 09 21:34:21.250131 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.250098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7qj82"]
Sep 09 21:34:21.252792 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:21.252761    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc86fd8d_a0d6_44a0_b8ab_610aa64edcca.slice/crio-bc56323b9907dfa4987f01608341fa1a8b37f8f948ab9c731c348ca057850e82 WatchSource:0}: Error finding container bc56323b9907dfa4987f01608341fa1a8b37f8f948ab9c731c348ca057850e82: Status 404 returned error can't find the container with id bc56323b9907dfa4987f01608341fa1a8b37f8f948ab9c731c348ca057850e82
Sep 09 21:34:21.384081 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.384050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7qj82" event={"ID":"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca","Type":"ContainerStarted","Data":"1a9fe072baa3941b10fadac08e3401a6beb1c222995e62f200cb90dbda86a941"}
Sep 09 21:34:21.384271 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.384104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7qj82" event={"ID":"cc86fd8d-a0d6-44a0-b8ab-610aa64edcca","Type":"ContainerStarted","Data":"bc56323b9907dfa4987f01608341fa1a8b37f8f948ab9c731c348ca057850e82"}
Sep 09 21:34:21.384271 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.384245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:34:21.385272 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.385250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7dlnc" event={"ID":"f85f5db0-2786-426b-a8dc-b6f7d6f37fed","Type":"ContainerStarted","Data":"bc11dd925cd8950e407613d34c4c542f5e4ab432696a9a45ae0ca19a74e5ecac"}
Sep 09 21:34:21.399011 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.398964    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7qj82" podStartSLOduration=66.398951049 podStartE2EDuration="1m6.398951049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:21.3983114 +0000 UTC m=+66.884194434" watchObservedRunningTime="2026-09-09 21:34:21.398951049 +0000 UTC m=+66.884834087"
Sep 09 21:34:21.473888 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.473808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4v4f\""
Sep 09 21:34:21.482069 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.482042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cplkq"
Sep 09 21:34:21.525328 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.525293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:21.525497 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.525332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:21.528763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.528636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"image-registry-cc4878bb4-sfpmd\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") " pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:21.528998 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.528969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8ppsw\" (UID: \"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:21.624912 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.624878    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ts5sm\""
Sep 09 21:34:21.633029 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.633009    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:21.749269 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.749183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cplkq"]
Sep 09 21:34:21.753063 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:21.753033    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70d3852_c3a8_4a65_b4bc_661f6a30a9d4.slice/crio-0ccc3cdb32512eb1cb29773b42a64d0f763af71c7e268ceb2522a01e018e4e75 WatchSource:0}: Error finding container 0ccc3cdb32512eb1cb29773b42a64d0f763af71c7e268ceb2522a01e018e4e75: Status 404 returned error can't find the container with id 0ccc3cdb32512eb1cb29773b42a64d0f763af71c7e268ceb2522a01e018e4e75
Sep 09 21:34:21.829466 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.829428    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r9ssj\""
Sep 09 21:34:21.837607 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.837580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"
Sep 09 21:34:21.910006 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:21.909972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc4878bb4-sfpmd"]
Sep 09 21:34:21.913817 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:21.913787    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b294a07_8857_420a_bfec_80269ffaf9bc.slice/crio-4b8e798344ae390005a3a8ae621e5828fdc95f4acdc9615c85e2f6f7b080562f WatchSource:0}: Error finding container 4b8e798344ae390005a3a8ae621e5828fdc95f4acdc9615c85e2f6f7b080562f: Status 404 returned error can't find the container with id 4b8e798344ae390005a3a8ae621e5828fdc95f4acdc9615c85e2f6f7b080562f
Sep 09 21:34:22.118224 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:22.118198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw"]
Sep 09 21:34:22.390658 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:22.390569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd" event={"ID":"3b294a07-8857-420a-bfec-80269ffaf9bc","Type":"ContainerStarted","Data":"dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d"}
Sep 09 21:34:22.390658 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:22.390628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd" event={"ID":"3b294a07-8857-420a-bfec-80269ffaf9bc","Type":"ContainerStarted","Data":"4b8e798344ae390005a3a8ae621e5828fdc95f4acdc9615c85e2f6f7b080562f"}
Sep 09 21:34:22.390908 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:22.390865    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:22.392565 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:22.392498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw" event={"ID":"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b","Type":"ContainerStarted","Data":"daa231ad80bc70819c9480e083144d27b10b9d8643b840980a4bb4f273ac2693"}
Sep 09 21:34:22.394428 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:22.394397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cplkq" event={"ID":"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4","Type":"ContainerStarted","Data":"0ccc3cdb32512eb1cb29773b42a64d0f763af71c7e268ceb2522a01e018e4e75"}
Sep 09 21:34:22.410943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:22.410557    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd" podStartSLOduration=17.410535294 podStartE2EDuration="17.410535294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:22.409170047 +0000 UTC m=+67.895053087" watchObservedRunningTime="2026-09-09 21:34:22.410535294 +0000 UTC m=+67.896418334"
Sep 09 21:34:26.409082 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.409040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw" event={"ID":"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b","Type":"ContainerStarted","Data":"14c44d48178b9d9d7ae5d5d9aef54dae5d1d4aca70fa2b916add0d1d0a27a182"}
Sep 09 21:34:26.409530 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.409099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw" event={"ID":"7d1ebe6a-efbe-4849-b0d4-3bf37ac1861b","Type":"ContainerStarted","Data":"b12fa2d84dfc3c92f7cd37d064940ebff655397b101183e8bae83dd6b8fb71f2"}
Sep 09 21:34:26.410928 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.410904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cplkq" event={"ID":"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4","Type":"ContainerStarted","Data":"19775dd5088ab8d6503f8e237376dddb631845267c65d93104aa9b8c8ac9df43"}
Sep 09 21:34:26.411052 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.410951    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cplkq"
Sep 09 21:34:26.411052 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.410967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cplkq" event={"ID":"c70d3852-c3a8-4a65-b4bc-661f6a30a9d4","Type":"ContainerStarted","Data":"ff37da51b1a6ed6783f37c1b5eed96a9a020ddd334ed13e422e5ed1a10e1efc6"}
Sep 09 21:34:26.412348 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.412324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7dlnc" event={"ID":"f85f5db0-2786-426b-a8dc-b6f7d6f37fed","Type":"ContainerStarted","Data":"abead05c4e7b456492f3a983263b7cb3e1a6be23cea148d6af2a721577b37843"}
Sep 09 21:34:26.429002 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.428960    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8ppsw" podStartSLOduration=17.898290441 podStartE2EDuration="21.428945886s" totalImagesPullingTime="3.530655445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:26.428052883 +0000 UTC m=+71.913935915" watchObservedRunningTime="2026-09-09 21:34:26.428945886 +0000 UTC m=+71.914828924"
Sep 09 21:34:26.442253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.442202    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7dlnc" podStartSLOduration=66.958661145 podStartE2EDuration="1m11.44218504s" totalImagesPullingTime="4.483523895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:26.441993716 +0000 UTC m=+71.927876757" watchObservedRunningTime="2026-09-09 21:34:26.44218504 +0000 UTC m=+71.928068078"
Sep 09 21:34:26.459816 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:26.459773    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cplkq" podStartSLOduration=33.516077444 podStartE2EDuration="37.459759198s" totalImagesPullingTime="3.943681754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:26.458792634 +0000 UTC m=+71.944675672" watchObservedRunningTime="2026-09-09 21:34:26.459759198 +0000 UTC m=+71.945642235"
Sep 09 21:34:27.471978 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:27.471936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:27.472466 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:27.472003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:27.472577 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:27.472555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9848c116-2d5e-4afd-abad-74193bc624cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:27.474410 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:27.474392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9848c116-2d5e-4afd-abad-74193bc624cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxrq\" (UID: \"9848c116-2d5e-4afd-abad-74193bc624cf\") " pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:27.520794 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:27.520763    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b8s76\""
Sep 09 21:34:27.528952 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:27.528928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mxrq"
Sep 09 21:34:27.829770 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:27.829744    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxrq"]
Sep 09 21:34:27.832206 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:27.832180    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9848c116_2d5e_4afd_abad_74193bc624cf.slice/crio-6a24bd7ee1bca83fbf93135b8b2677b093ea3a1d44779ff78559b9293507a247 WatchSource:0}: Error finding container 6a24bd7ee1bca83fbf93135b8b2677b093ea3a1d44779ff78559b9293507a247: Status 404 returned error can't find the container with id 6a24bd7ee1bca83fbf93135b8b2677b093ea3a1d44779ff78559b9293507a247
Sep 09 21:34:28.421285 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:28.421250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxrq" event={"ID":"9848c116-2d5e-4afd-abad-74193bc624cf","Type":"ContainerStarted","Data":"4c18eb0efe78e690fe39f02d3b55ee4e7d1b38cec4d641c8fa247a3f59e5d793"}
Sep 09 21:34:28.421456 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:28.421306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxrq" event={"ID":"9848c116-2d5e-4afd-abad-74193bc624cf","Type":"ContainerStarted","Data":"6a24bd7ee1bca83fbf93135b8b2677b093ea3a1d44779ff78559b9293507a247"}
Sep 09 21:34:29.426194 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:29.426157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxrq" event={"ID":"9848c116-2d5e-4afd-abad-74193bc624cf","Type":"ContainerStarted","Data":"8ec68bec95c66aec4eaf4f14f7cb5f759f4e6f951ce87dba7a63fc41b71adafb"}
Sep 09 21:34:31.437846 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:31.437805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxrq" event={"ID":"9848c116-2d5e-4afd-abad-74193bc624cf","Type":"ContainerStarted","Data":"66bf09f6d3e3a3b4a478459fd7374e2af9e83e0f5d2baa00c29f3609251d0ee6"}
Sep 09 21:34:31.457938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:31.457861    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8mxrq" podStartSLOduration=17.980186742 podStartE2EDuration="20.45784388s" totalImagesPullingTime="2.477657138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:34:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:31.456768304 +0000 UTC m=+76.942651345" watchObservedRunningTime="2026-09-09 21:34:31.45784388 +0000 UTC m=+76.943726917"
Sep 09 21:34:34.824588 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.824556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"]
Sep 09 21:34:34.827315 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.827299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"
Sep 09 21:34:34.831340 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.831306    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 21:34:34.831340 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.831327    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 21:34:34.831480 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.831310    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 21:34:34.831480 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.831362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 21:34:34.831480 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.831403    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4vbn2\""
Sep 09 21:34:34.842424 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.842401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"]
Sep 09 21:34:34.868084 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.868052    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cc4878bb4-sfpmd"]
Sep 09 21:34:34.914538 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.914504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"]
Sep 09 21:34:34.917264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.917249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:34.926957 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.926934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09d8122d-0df8-4911-ab8e-6a4f6bbb8a51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cfffd476d-rvbnj\" (UID: \"09d8122d-0df8-4911-ab8e-6a4f6bbb8a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"
Sep 09 21:34:34.927049 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.926973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zhd\" (UniqueName: \"kubernetes.io/projected/09d8122d-0df8-4911-ab8e-6a4f6bbb8a51-kube-api-access-q6zhd\") pod \"managed-serviceaccount-addon-agent-6cfffd476d-rvbnj\" (UID: \"09d8122d-0df8-4911-ab8e-6a4f6bbb8a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"
Sep 09 21:34:34.945018 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.944992    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-6q66v"]
Sep 09 21:34:34.948321 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.948300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6q66v"
Sep 09 21:34:34.948433 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.948311    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"]
Sep 09 21:34:34.951082 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.951064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kn82b\""
Sep 09 21:34:34.951198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.951183    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 21:34:34.951329 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.951308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 21:34:34.963942 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:34.963920    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6q66v"]
Sep 09 21:34:35.027985 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.027946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-registry-tls\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.027985 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.027992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-registry-certificates\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.028215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-trusted-ca\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.028215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zhd\" (UniqueName: \"kubernetes.io/projected/09d8122d-0df8-4911-ab8e-6a4f6bbb8a51-kube-api-access-q6zhd\") pod \"managed-serviceaccount-addon-agent-6cfffd476d-rvbnj\" (UID: \"09d8122d-0df8-4911-ab8e-6a4f6bbb8a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"
Sep 09 21:34:35.028215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-ca-trust-extracted\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.028215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-installation-pull-secrets\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.028215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09d8122d-0df8-4911-ab8e-6a4f6bbb8a51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cfffd476d-rvbnj\" (UID: \"09d8122d-0df8-4911-ab8e-6a4f6bbb8a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"
Sep 09 21:34:35.028215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9npz\" (UniqueName: \"kubernetes.io/projected/44c1323a-df33-4e95-a420-f5890fa482a8-kube-api-access-h9npz\") pod \"downloads-85d98c998f-6q66v\" (UID: \"44c1323a-df33-4e95-a420-f5890fa482a8\") " pod="openshift-console/downloads-85d98c998f-6q66v"
Sep 09 21:34:35.028215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-image-registry-private-configuration\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.028215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-bound-sa-token\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.028563 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.028242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw6hl\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-kube-api-access-mw6hl\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.030785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.030757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09d8122d-0df8-4911-ab8e-6a4f6bbb8a51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cfffd476d-rvbnj\" (UID: \"09d8122d-0df8-4911-ab8e-6a4f6bbb8a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"
Sep 09 21:34:35.038671 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.038644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zhd\" (UniqueName: \"kubernetes.io/projected/09d8122d-0df8-4911-ab8e-6a4f6bbb8a51-kube-api-access-q6zhd\") pod \"managed-serviceaccount-addon-agent-6cfffd476d-rvbnj\" (UID: \"09d8122d-0df8-4911-ab8e-6a4f6bbb8a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"
Sep 09 21:34:35.128599 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-installation-pull-secrets\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.128785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9npz\" (UniqueName: \"kubernetes.io/projected/44c1323a-df33-4e95-a420-f5890fa482a8-kube-api-access-h9npz\") pod \"downloads-85d98c998f-6q66v\" (UID: \"44c1323a-df33-4e95-a420-f5890fa482a8\") " pod="openshift-console/downloads-85d98c998f-6q66v"
Sep 09 21:34:35.128785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-image-registry-private-configuration\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.128785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-bound-sa-token\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.128785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw6hl\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-kube-api-access-mw6hl\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.128785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-registry-tls\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.129029 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-registry-certificates\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.129029 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-trusted-ca\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.129029 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.128871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-ca-trust-extracted\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.129369 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.129290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-ca-trust-extracted\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.129845 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.129820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-registry-certificates\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.129955 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.129846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-trusted-ca\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.131285 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.131261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-installation-pull-secrets\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.131378 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.131338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-registry-tls\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.131413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.131389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-image-registry-private-configuration\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.139472 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.139452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw6hl\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-kube-api-access-mw6hl\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.139547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.139458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9npz\" (UniqueName: \"kubernetes.io/projected/44c1323a-df33-4e95-a420-f5890fa482a8-kube-api-access-h9npz\") pod \"downloads-85d98c998f-6q66v\" (UID: \"44c1323a-df33-4e95-a420-f5890fa482a8\") " pod="openshift-console/downloads-85d98c998f-6q66v"
Sep 09 21:34:35.139648 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.139629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b832e8f5-9fbb-4acd-869a-9a47f01d15ad-bound-sa-token\") pod \"image-registry-7d4fccf9d6-s9bnt\" (UID: \"b832e8f5-9fbb-4acd-869a-9a47f01d15ad\") " pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.155471 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.155430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"
Sep 09 21:34:35.227209 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.226886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:35.257345 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.257309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6q66v"
Sep 09 21:34:35.421271 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.421201    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj"]
Sep 09 21:34:35.424402 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:35.424376    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d8122d_0df8_4911_ab8e_6a4f6bbb8a51.slice/crio-29439a18af0f2d247e15b2a75ecc5f16c92d08be76de1e9d0ad8558ecd795d13 WatchSource:0}: Error finding container 29439a18af0f2d247e15b2a75ecc5f16c92d08be76de1e9d0ad8558ecd795d13: Status 404 returned error can't find the container with id 29439a18af0f2d247e15b2a75ecc5f16c92d08be76de1e9d0ad8558ecd795d13
Sep 09 21:34:35.450192 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.450082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj" event={"ID":"09d8122d-0df8-4911-ab8e-6a4f6bbb8a51","Type":"ContainerStarted","Data":"29439a18af0f2d247e15b2a75ecc5f16c92d08be76de1e9d0ad8558ecd795d13"}
Sep 09 21:34:35.515655 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.515631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"]
Sep 09 21:34:35.518063 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:35.518029    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb832e8f5_9fbb_4acd_869a_9a47f01d15ad.slice/crio-7c7724660645ab25afffecb05bdda91a0e0413b5466949abe9e39584e4541400 WatchSource:0}: Error finding container 7c7724660645ab25afffecb05bdda91a0e0413b5466949abe9e39584e4541400: Status 404 returned error can't find the container with id 7c7724660645ab25afffecb05bdda91a0e0413b5466949abe9e39584e4541400
Sep 09 21:34:35.532379 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:35.532359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6q66v"]
Sep 09 21:34:35.534977 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:35.534952    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c1323a_df33_4e95_a420_f5890fa482a8.slice/crio-67bf8d089f384cbe255a358bee59871a342887f292effe3bebc8e4f72c7570e3 WatchSource:0}: Error finding container 67bf8d089f384cbe255a358bee59871a342887f292effe3bebc8e4f72c7570e3: Status 404 returned error can't find the container with id 67bf8d089f384cbe255a358bee59871a342887f292effe3bebc8e4f72c7570e3
Sep 09 21:34:36.418778 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:36.418748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cplkq"
Sep 09 21:34:36.456131 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:36.456090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6q66v" event={"ID":"44c1323a-df33-4e95-a420-f5890fa482a8","Type":"ContainerStarted","Data":"67bf8d089f384cbe255a358bee59871a342887f292effe3bebc8e4f72c7570e3"}
Sep 09 21:34:36.458617 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:36.458590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt" event={"ID":"b832e8f5-9fbb-4acd-869a-9a47f01d15ad","Type":"ContainerStarted","Data":"13644657ec1f5800b601568a19c0c093d62a14702b1d05c195b533307084e2e8"}
Sep 09 21:34:36.458776 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:36.458636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt" event={"ID":"b832e8f5-9fbb-4acd-869a-9a47f01d15ad","Type":"ContainerStarted","Data":"7c7724660645ab25afffecb05bdda91a0e0413b5466949abe9e39584e4541400"}
Sep 09 21:34:36.459262 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:36.459240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:36.480739 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:36.480669    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt" podStartSLOduration=2.480647909 podStartE2EDuration="2.480647909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:36.480180375 +0000 UTC m=+81.966063443" watchObservedRunningTime="2026-09-09 21:34:36.480647909 +0000 UTC m=+81.966530940"
Sep 09 21:34:38.240815 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.240700    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78d9668758-nqmgs"]
Sep 09 21:34:38.244023 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.243997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.246446 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.246419    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 21:34:38.246564 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.246512    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 21:34:38.247925 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.247803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v2kxg\""
Sep 09 21:34:38.247925 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.247814    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 21:34:38.247925 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.247803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 21:34:38.247925 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.247808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 21:34:38.255038 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.255018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d9668758-nqmgs"]
Sep 09 21:34:38.357918 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.357887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-config\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.358116 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.357934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-serving-cert\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.358116 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.358035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-service-ca\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.358116 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.358076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-oauth-config\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.358258 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.358145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-oauth-serving-cert\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.358258 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.358193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fsf\" (UniqueName: \"kubernetes.io/projected/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-kube-api-access-h2fsf\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.458788 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.458748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-config\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.458919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.458812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-serving-cert\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.458919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.458881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-service-ca\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.458919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.458907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-oauth-config\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.459071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.458941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-oauth-serving-cert\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.459071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.458968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fsf\" (UniqueName: \"kubernetes.io/projected/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-kube-api-access-h2fsf\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.459454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.459431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-config\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.460332 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.460280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-service-ca\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.460985 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.460958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-oauth-serving-cert\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.462087 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.462064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-oauth-config\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.462312 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.462289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-serving-cert\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.467906 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.467883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fsf\" (UniqueName: \"kubernetes.io/projected/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-kube-api-access-h2fsf\") pod \"console-78d9668758-nqmgs\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") " pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.557583 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.557489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:38.837204 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:38.837128    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d9668758-nqmgs"]
Sep 09 21:34:38.841931 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:38.841907    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc25d7b_d1ac_444f_b3f3_8147ac964dec.slice/crio-9af502e21157b816679f81b6e170c2abcf80a6d39e3d22d95499da20d6ec6b15 WatchSource:0}: Error finding container 9af502e21157b816679f81b6e170c2abcf80a6d39e3d22d95499da20d6ec6b15: Status 404 returned error can't find the container with id 9af502e21157b816679f81b6e170c2abcf80a6d39e3d22d95499da20d6ec6b15
Sep 09 21:34:39.473201 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:39.473140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj" event={"ID":"09d8122d-0df8-4911-ab8e-6a4f6bbb8a51","Type":"ContainerStarted","Data":"9bb70c6128d65a45aef6e129db4d084e8891aab2bf39e99c7c67b4e236619785"}
Sep 09 21:34:39.474823 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:39.474789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d9668758-nqmgs" event={"ID":"0bc25d7b-d1ac-444f-b3f3-8147ac964dec","Type":"ContainerStarted","Data":"9af502e21157b816679f81b6e170c2abcf80a6d39e3d22d95499da20d6ec6b15"}
Sep 09 21:34:39.490161 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:39.490077    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfffd476d-rvbnj" podStartSLOduration=2.513626408 podStartE2EDuration="5.490058256s" totalImagesPullingTime="2.976431848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:39.489235315 +0000 UTC m=+84.975118356" watchObservedRunningTime="2026-09-09 21:34:39.490058256 +0000 UTC m=+84.975941291"
Sep 09 21:34:43.493447 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:43.493401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d9668758-nqmgs" event={"ID":"0bc25d7b-d1ac-444f-b3f3-8147ac964dec","Type":"ContainerStarted","Data":"3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a"}
Sep 09 21:34:43.511640 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:43.511578    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78d9668758-nqmgs" podStartSLOduration=1.5463244710000001 podStartE2EDuration="5.511561691s" totalImagesPullingTime="3.96523722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:43.509623404 +0000 UTC m=+88.995506445" watchObservedRunningTime="2026-09-09 21:34:43.511561691 +0000 UTC m=+88.997444802"
Sep 09 21:34:44.874051 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:44.874022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:34:46.922520 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:46.922487    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6448c9c59d-8qxxp"]
Sep 09 21:34:46.924987 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:46.924967    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:46.934288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:46.934268    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 21:34:46.936664 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:46.936640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6448c9c59d-8qxxp"]
Sep 09 21:34:47.035563 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.035519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-oauth-serving-cert\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.035761 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.035578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-trusted-ca-bundle\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.035761 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.035668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-oauth-config\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.035761 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.035710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2cpj\" (UniqueName: \"kubernetes.io/projected/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-kube-api-access-h2cpj\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.035905 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.035817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-serving-cert\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.035905 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.035847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-service-ca\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.035973 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.035903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-config\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.136342 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.136292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-config\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.136342 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.136334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-oauth-serving-cert\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.136582 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.136365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-trusted-ca-bundle\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.136582 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.136411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-oauth-config\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.136582 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.136435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2cpj\" (UniqueName: \"kubernetes.io/projected/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-kube-api-access-h2cpj\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.136582 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.136504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-serving-cert\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.136582 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.136533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-service-ca\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.137176 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.137145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-config\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.137176 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.137169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-oauth-serving-cert\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.137521 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.137269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-trusted-ca-bundle\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.137521 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.137284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-service-ca\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.139266 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.139242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-oauth-config\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.139383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.139358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-serving-cert\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.144314 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.144292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2cpj\" (UniqueName: \"kubernetes.io/projected/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-kube-api-access-h2cpj\") pod \"console-6448c9c59d-8qxxp\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") " pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:47.235695 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:47.235605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:48.557943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:48.557907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:48.558354 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:48.557995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:48.563393 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:48.563371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:49.519824 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:49.519796    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:34:52.396667 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:52.396639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7qj82"
Sep 09 21:34:52.528120 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:52.528090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6q66v" event={"ID":"44c1323a-df33-4e95-a420-f5890fa482a8","Type":"ContainerStarted","Data":"bca7f049ddbcd77884e76c765bd01fb32777c1a7fdd496a0ddc6511425463844"}
Sep 09 21:34:52.528307 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:52.528271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-6q66v"
Sep 09 21:34:52.529700 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:52.529661    2563 patch_prober.go:28] interesting pod/downloads-85d98c998f-6q66v container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" start-of-body=
Sep 09 21:34:52.529834 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:52.529803    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-6q66v" podUID="44c1323a-df33-4e95-a420-f5890fa482a8" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused"
Sep 09 21:34:52.548826 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:52.548770    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-6q66v" podStartSLOduration=1.6673862499999998 podStartE2EDuration="18.548750062s" totalImagesPullingTime="16.881363812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:52.547742151 +0000 UTC m=+98.033625182" watchObservedRunningTime="2026-09-09 21:34:52.548750062 +0000 UTC m=+98.034633098"
Sep 09 21:34:52.587581 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:52.587475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6448c9c59d-8qxxp"]
Sep 09 21:34:52.590472 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:52.590437    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647c4f0a_1eea_4a84_8900_47b5cf7f88d2.slice/crio-7ef3536cd8e038b562e10f2eaeaa8be1910072b093efecd78f98a9b61b34e505 WatchSource:0}: Error finding container 7ef3536cd8e038b562e10f2eaeaa8be1910072b093efecd78f98a9b61b34e505: Status 404 returned error can't find the container with id 7ef3536cd8e038b562e10f2eaeaa8be1910072b093efecd78f98a9b61b34e505
Sep 09 21:34:53.218291 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.218256    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"]
Sep 09 21:34:53.221906 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.221873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.227707 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.227680    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 21:34:53.227999 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.227981    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 21:34:53.230820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.230780    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 21:34:53.230820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.230814    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 21:34:53.230989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.230853    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 21:34:53.230989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.230965    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kvww8\""
Sep 09 21:34:53.231135 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.231115    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 21:34:53.234554 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.233066    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gcmmm"]
Sep 09 21:34:53.236646 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.236627    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"]
Sep 09 21:34:53.236790 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.236758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.239392 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.239370    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 21:34:53.239661 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.239460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 21:34:53.239950 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.239912    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 21:34:53.240234 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.239972    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9tb2f\""
Sep 09 21:34:53.391356 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.391356 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-sys\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.391586 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.391586 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73bca816-b6c0-43e0-bd86-4226b7b91e7c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.391586 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.391586 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.391586 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-root\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.391586 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73bca816-b6c0-43e0-bd86-4226b7b91e7c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.391933 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-wtmp\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.391933 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk24t\" (UniqueName: \"kubernetes.io/projected/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-kube-api-access-zk24t\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.391933 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lsnd\" (UniqueName: \"kubernetes.io/projected/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-api-access-4lsnd\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.391933 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-metrics-client-ca\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.391933 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-tls\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.391933 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-textfile\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.392190 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.391942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.493469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.493469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73bca816-b6c0-43e0-bd86-4226b7b91e7c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.493469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-root\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73bca816-b6c0-43e0-bd86-4226b7b91e7c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-wtmp\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk24t\" (UniqueName: \"kubernetes.io/projected/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-kube-api-access-zk24t\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-root\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lsnd\" (UniqueName: \"kubernetes.io/projected/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-api-access-4lsnd\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-metrics-client-ca\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-tls\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-textfile\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.493915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-sys\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.494017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-sys\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494850 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.494235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73bca816-b6c0-43e0-bd86-4226b7b91e7c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.494850 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.494270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73bca816-b6c0-43e0-bd86-4226b7b91e7c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.494850 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.494352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-wtmp\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.494850 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.494479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.495074 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.494935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-metrics-client-ca\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.495074 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.495030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.495272 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.495251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-textfile\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.496561 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.496417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-tls\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.496561 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.496525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.496804 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.496784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.497099 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.497081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.516883 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.516858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk24t\" (UniqueName: \"kubernetes.io/projected/9b72e7a2-18db-4d9d-b0c4-6eb850c905a6-kube-api-access-zk24t\") pod \"node-exporter-gcmmm\" (UID: \"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6\") " pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.522967 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.522940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lsnd\" (UniqueName: \"kubernetes.io/projected/73bca816-b6c0-43e0-bd86-4226b7b91e7c-kube-api-access-4lsnd\") pod \"kube-state-metrics-74cb57df7-bs9rc\" (UID: \"73bca816-b6c0-43e0-bd86-4226b7b91e7c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.533676 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.533652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"
Sep 09 21:34:53.534191 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.534148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6448c9c59d-8qxxp" event={"ID":"647c4f0a-1eea-4a84-8900-47b5cf7f88d2","Type":"ContainerStarted","Data":"46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c"}
Sep 09 21:34:53.534306 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.534207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6448c9c59d-8qxxp" event={"ID":"647c4f0a-1eea-4a84-8900-47b5cf7f88d2","Type":"ContainerStarted","Data":"7ef3536cd8e038b562e10f2eaeaa8be1910072b093efecd78f98a9b61b34e505"}
Sep 09 21:34:53.548186 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.548162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gcmmm"
Sep 09 21:34:53.549871 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.549844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-6q66v"
Sep 09 21:34:53.555307 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.554098    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6448c9c59d-8qxxp" podStartSLOduration=7.554078446 podStartE2EDuration="7.554078446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:53.553073612 +0000 UTC m=+99.038956676" watchObservedRunningTime="2026-09-09 21:34:53.554078446 +0000 UTC m=+99.039961481"
Sep 09 21:34:53.564780 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:53.564701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b72e7a2_18db_4d9d_b0c4_6eb850c905a6.slice/crio-97e74a378f849f4a22d8694f5e503bf8675ba20686b53a3d4758b02a45ba1c93 WatchSource:0}: Error finding container 97e74a378f849f4a22d8694f5e503bf8675ba20686b53a3d4758b02a45ba1c93: Status 404 returned error can't find the container with id 97e74a378f849f4a22d8694f5e503bf8675ba20686b53a3d4758b02a45ba1c93
Sep 09 21:34:53.828932 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:53.828903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc"]
Sep 09 21:34:53.831585 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:53.831557    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bca816_b6c0_43e0_bd86_4226b7b91e7c.slice/crio-650b7f5b898f9cfe0e8973b2507d63e6189999efd501639b093e92a806063b02 WatchSource:0}: Error finding container 650b7f5b898f9cfe0e8973b2507d63e6189999efd501639b093e92a806063b02: Status 404 returned error can't find the container with id 650b7f5b898f9cfe0e8973b2507d63e6189999efd501639b093e92a806063b02
Sep 09 21:34:54.283182 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.283150    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 21:34:54.314992 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.314956    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 21:34:54.315160 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.315147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.319798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.318103    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 21:34:54.319798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.318156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 21:34:54.319798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.318100    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 21:34:54.319798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.318525    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 21:34:54.321284 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.321157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 21:34:54.321284 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.321245    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ngcsw\""
Sep 09 21:34:54.321832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.321654    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 21:34:54.322932 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.322290    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 21:34:54.322932 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.322511    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 21:34:54.322932 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.322702    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 21:34:54.402924 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.402740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.402924 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.402800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.402924 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.402819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t6fj\" (UniqueName: \"kubernetes.io/projected/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-kube-api-access-5t6fj\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.402924 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.402838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-config-out\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.402924 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.402853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.402924 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.402872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.403413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.402972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-web-config\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.403413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.403007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.403413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.403028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.403413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.403263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.403413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.403292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.403413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.403345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.403413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.403387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.503985 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.503944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.503997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t6fj\" (UniqueName: \"kubernetes.io/projected/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-kube-api-access-5t6fj\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-config-out\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-web-config\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.504425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.505050 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.505050 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.504769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.506643 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.506577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.506643 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.506601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.511359 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.510975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.511359 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.511315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.511942 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.511900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.512515 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.512455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-config-out\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.512515 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.512474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.512656 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.512600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.512871 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.512849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.513538 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.513514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-web-config\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.514528 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.514488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t6fj\" (UniqueName: \"kubernetes.io/projected/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-kube-api-access-5t6fj\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.515334 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.515307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0fe9a7f-5caf-4e35-848e-b9d07ed13697-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0fe9a7f-5caf-4e35-848e-b9d07ed13697\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:54.540459 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.540381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcmmm" event={"ID":"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6","Type":"ContainerStarted","Data":"97e74a378f849f4a22d8694f5e503bf8675ba20686b53a3d4758b02a45ba1c93"}
Sep 09 21:34:54.543421 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.542842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc" event={"ID":"73bca816-b6c0-43e0-bd86-4226b7b91e7c","Type":"ContainerStarted","Data":"650b7f5b898f9cfe0e8973b2507d63e6189999efd501639b093e92a806063b02"}
Sep 09 21:34:54.631293 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:54.631240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:34:55.235947 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.235921    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 21:34:55.299215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.299172    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"]
Sep 09 21:34:55.320493 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.320458    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"]
Sep 09 21:34:55.320663 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.320570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.323682 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.323656    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pkgt9\""
Sep 09 21:34:55.323846 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.323693    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 21:34:55.323846 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.323790    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 21:34:55.324158 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.324137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 21:34:55.324158 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.324151    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 21:34:55.324492 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.324472    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fmh0t4upcvqu4\""
Sep 09 21:34:55.324687 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.324669    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 21:34:55.415248 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.415206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.415442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.415279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.415442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.415313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c20acab5-48d1-46c0-9d06-6f03623f6987-metrics-client-ca\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.415442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.415334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-grpc-tls\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.415442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.415395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-tls\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.415661 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.415443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.415661 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.415475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.415661 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.415501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9x9\" (UniqueName: \"kubernetes.io/projected/c20acab5-48d1-46c0-9d06-6f03623f6987-kube-api-access-vw9x9\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.516567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.516115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.516567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.516173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.516567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.516203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9x9\" (UniqueName: \"kubernetes.io/projected/c20acab5-48d1-46c0-9d06-6f03623f6987-kube-api-access-vw9x9\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.516567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.516250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.516567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.516283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.516567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.516304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c20acab5-48d1-46c0-9d06-6f03623f6987-metrics-client-ca\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.516567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.516332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-grpc-tls\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.516567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.516391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-tls\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.518902 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.517917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c20acab5-48d1-46c0-9d06-6f03623f6987-metrics-client-ca\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.520130 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.519866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.520130 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.519955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.522421 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.522388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.522573 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.522545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.523276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.523248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-grpc-tls\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.524792 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.524749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c20acab5-48d1-46c0-9d06-6f03623f6987-secret-thanos-querier-tls\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.525932 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.525887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9x9\" (UniqueName: \"kubernetes.io/projected/c20acab5-48d1-46c0-9d06-6f03623f6987-kube-api-access-vw9x9\") pod \"thanos-querier-55d7cb6886-pp7s4\" (UID: \"c20acab5-48d1-46c0-9d06-6f03623f6987\") " pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.632920 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:55.632880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:34:55.808891 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:55.808823    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fe9a7f_5caf_4e35_848e_b9d07ed13697.slice/crio-b8de4e86e7f0f3cfd1b9906608b41454ccee55d1725b3366280d7ede7282621e WatchSource:0}: Error finding container b8de4e86e7f0f3cfd1b9906608b41454ccee55d1725b3366280d7ede7282621e: Status 404 returned error can't find the container with id b8de4e86e7f0f3cfd1b9906608b41454ccee55d1725b3366280d7ede7282621e
Sep 09 21:34:56.117981 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.117951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"]
Sep 09 21:34:56.122437 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:34:56.122403    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20acab5_48d1_46c0_9d06_6f03623f6987.slice/crio-45c33b7583e47201de9199a541710745093c11915e38b531f1e5351c24e53e3b WatchSource:0}: Error finding container 45c33b7583e47201de9199a541710745093c11915e38b531f1e5351c24e53e3b: Status 404 returned error can't find the container with id 45c33b7583e47201de9199a541710745093c11915e38b531f1e5351c24e53e3b
Sep 09 21:34:56.565283 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.564164    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b72e7a2-18db-4d9d-b0c4-6eb850c905a6" containerID="e2d16f8a392f057eb9a7cbb7546efb2db09cb4ea9b434b3a8ac7827525b07fac" exitCode=0
Sep 09 21:34:56.565283 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.564200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcmmm" event={"ID":"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6","Type":"ContainerDied","Data":"e2d16f8a392f057eb9a7cbb7546efb2db09cb4ea9b434b3a8ac7827525b07fac"}
Sep 09 21:34:56.567225 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.567186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0fe9a7f-5caf-4e35-848e-b9d07ed13697","Type":"ContainerStarted","Data":"b8de4e86e7f0f3cfd1b9906608b41454ccee55d1725b3366280d7ede7282621e"}
Sep 09 21:34:56.573131 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.573101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc" event={"ID":"73bca816-b6c0-43e0-bd86-4226b7b91e7c","Type":"ContainerStarted","Data":"8f3cd3c4c33e600eeb3968f11ec60e1f3ce12fbe6f22baf65f53bdc85a00b81f"}
Sep 09 21:34:56.573257 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.573160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc" event={"ID":"73bca816-b6c0-43e0-bd86-4226b7b91e7c","Type":"ContainerStarted","Data":"3af20b8f2561f9e38505e5f87aae298669f64e9b5374c0a00be170e968b435d7"}
Sep 09 21:34:56.573257 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.573188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc" event={"ID":"73bca816-b6c0-43e0-bd86-4226b7b91e7c","Type":"ContainerStarted","Data":"6bebbfde8621377ad41f692d4f3a4b9a7df9333a3e2e379a26e327c37d243869"}
Sep 09 21:34:56.575411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.575388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4" event={"ID":"c20acab5-48d1-46c0-9d06-6f03623f6987","Type":"ContainerStarted","Data":"45c33b7583e47201de9199a541710745093c11915e38b531f1e5351c24e53e3b"}
Sep 09 21:34:56.606218 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:56.605867    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bs9rc" podStartSLOduration=1.332213047 podStartE2EDuration="3.605846607s" totalImagesPullingTime="2.27363356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:56.605058285 +0000 UTC m=+102.090941402" watchObservedRunningTime="2026-09-09 21:34:56.605846607 +0000 UTC m=+102.091729643"
Sep 09 21:34:57.236241 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:57.236197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:57.236430 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:57.236267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:57.242565 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:57.242534    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:57.587821 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:57.587651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcmmm" event={"ID":"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6","Type":"ContainerStarted","Data":"809e84fa50c7456ec737ce040660a0fd34d51234c7cd6e0e3956e957a8721844"}
Sep 09 21:34:57.587821 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:57.587731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcmmm" event={"ID":"9b72e7a2-18db-4d9d-b0c4-6eb850c905a6","Type":"ContainerStarted","Data":"21a9a52c736abc47c20227a1c2d08496e41c441b5aa4686b78bfd5021106af24"}
Sep 09 21:34:57.592388 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:57.592360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:34:57.609150 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:57.609098    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gcmmm" podStartSLOduration=2.249670819 podStartE2EDuration="4.609082093s" totalImagesPullingTime="2.267149678s" totalInitContainerRuntime="92.261596ms" isStatefulPod=true podCreationTimestamp="2026-09-09 21:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:57.608016921 +0000 UTC m=+103.093899959" watchObservedRunningTime="2026-09-09 21:34:57.609082093 +0000 UTC m=+103.094965122"
Sep 09 21:34:57.666711 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:57.666677    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d9668758-nqmgs"]
Sep 09 21:34:58.473993 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:58.473955    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d4fccf9d6-s9bnt"
Sep 09 21:34:58.595364 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:58.595327    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0fe9a7f-5caf-4e35-848e-b9d07ed13697" containerID="296c1d981d63855dcb58244c2301c8c885e812cece0ed23dbe714fb2cab27800" exitCode=0
Sep 09 21:34:58.596616 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:58.596590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0fe9a7f-5caf-4e35-848e-b9d07ed13697","Type":"ContainerDied","Data":"296c1d981d63855dcb58244c2301c8c885e812cece0ed23dbe714fb2cab27800"}
Sep 09 21:34:59.607173 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.607141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4" event={"ID":"c20acab5-48d1-46c0-9d06-6f03623f6987","Type":"ContainerStarted","Data":"9bf9fc5e346cee8bc8a200d8a4f88d8d7267db02bde3b340baf0f46f38725581"}
Sep 09 21:34:59.745475 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.743921    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c45ff568-gs674"]
Sep 09 21:34:59.765591 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.764885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c45ff568-gs674"]
Sep 09 21:34:59.765591 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.765011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.862254 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.862221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-console-config\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.862414 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.862299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-oauth-serving-cert\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.862414 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.862354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-oauth-config\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.862414 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.862410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmpzl\" (UniqueName: \"kubernetes.io/projected/62956573-ed81-4758-9a6f-493e609e299f-kube-api-access-mmpzl\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.862579 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.862479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-service-ca\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.862579 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.862550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-serving-cert\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.862673 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.862577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-trusted-ca-bundle\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.888493 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.888452    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd" podUID="3b294a07-8857-420a-bfec-80269ffaf9bc" containerName="registry" containerID="cri-o://dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d" gracePeriod=30
Sep 09 21:34:59.963328 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.963284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-oauth-serving-cert\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.963494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.963334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-oauth-config\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.963494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.963358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmpzl\" (UniqueName: \"kubernetes.io/projected/62956573-ed81-4758-9a6f-493e609e299f-kube-api-access-mmpzl\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.963494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.963418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-service-ca\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.963494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.963483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-serving-cert\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.963756 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.963504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-trusted-ca-bundle\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.963756 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.963548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-console-config\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.964327 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.964267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-oauth-serving-cert\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.964327 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.964314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-console-config\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.964598 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.964570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-trusted-ca-bundle\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.964775 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.964748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-service-ca\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.966493 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.966467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-serving-cert\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.966761 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.966687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-oauth-config\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:34:59.971471 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:34:59.971451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmpzl\" (UniqueName: \"kubernetes.io/projected/62956573-ed81-4758-9a6f-493e609e299f-kube-api-access-mmpzl\") pod \"console-5c45ff568-gs674\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") " pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:35:00.093362 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.092906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:35:00.209041 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.208863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:35:00.269387 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.269306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") pod \"3b294a07-8857-420a-bfec-80269ffaf9bc\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") "
Sep 09 21:35:00.269387 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.269375    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-installation-pull-secrets\") pod \"3b294a07-8857-420a-bfec-80269ffaf9bc\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") "
Sep 09 21:35:00.269614 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.269408    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-bound-sa-token\") pod \"3b294a07-8857-420a-bfec-80269ffaf9bc\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") "
Sep 09 21:35:00.269614 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.269455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-trusted-ca\") pod \"3b294a07-8857-420a-bfec-80269ffaf9bc\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") "
Sep 09 21:35:00.269614 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.269486    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b294a07-8857-420a-bfec-80269ffaf9bc-ca-trust-extracted\") pod \"3b294a07-8857-420a-bfec-80269ffaf9bc\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") "
Sep 09 21:35:00.269614 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.269525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-image-registry-private-configuration\") pod \"3b294a07-8857-420a-bfec-80269ffaf9bc\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") "
Sep 09 21:35:00.269614 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.269555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-certificates\") pod \"3b294a07-8857-420a-bfec-80269ffaf9bc\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") "
Sep 09 21:35:00.269614 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.269609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94lcb\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-kube-api-access-94lcb\") pod \"3b294a07-8857-420a-bfec-80269ffaf9bc\" (UID: \"3b294a07-8857-420a-bfec-80269ffaf9bc\") "
Sep 09 21:35:00.270873 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.270663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3b294a07-8857-420a-bfec-80269ffaf9bc" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:00.272613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.271389    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3b294a07-8857-420a-bfec-80269ffaf9bc" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:00.273859 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.273688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-kube-api-access-94lcb" (OuterVolumeSpecName: "kube-api-access-94lcb") pod "3b294a07-8857-420a-bfec-80269ffaf9bc" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc"). InnerVolumeSpecName "kube-api-access-94lcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:35:00.274100 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.274063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3b294a07-8857-420a-bfec-80269ffaf9bc" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:35:00.275472 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.275440    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3b294a07-8857-420a-bfec-80269ffaf9bc" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:35:00.275567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.275484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3b294a07-8857-420a-bfec-80269ffaf9bc" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:35:00.275893 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.275855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3b294a07-8857-420a-bfec-80269ffaf9bc" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:35:00.282346 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.282318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b294a07-8857-420a-bfec-80269ffaf9bc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3b294a07-8857-420a-bfec-80269ffaf9bc" (UID: "3b294a07-8857-420a-bfec-80269ffaf9bc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:35:00.370512 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.370474    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:00.370512 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.370512    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-installation-pull-secrets\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:00.370752 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.370527    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-bound-sa-token\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:00.370752 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.370539    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-trusted-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:00.370752 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.370550    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b294a07-8857-420a-bfec-80269ffaf9bc-ca-trust-extracted\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:00.370752 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.370562    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b294a07-8857-420a-bfec-80269ffaf9bc-image-registry-private-configuration\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:00.370752 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.370575    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b294a07-8857-420a-bfec-80269ffaf9bc-registry-certificates\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:00.370752 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.370587    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94lcb\" (UniqueName: \"kubernetes.io/projected/3b294a07-8857-420a-bfec-80269ffaf9bc-kube-api-access-94lcb\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:00.402033 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.402007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c45ff568-gs674"]
Sep 09 21:35:00.404232 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:35:00.404200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62956573_ed81_4758_9a6f_493e609e299f.slice/crio-7739ccca0bbcf85f60c56ef0dd2bee780b5ede268f6a8064ee9477f1cccca4f6 WatchSource:0}: Error finding container 7739ccca0bbcf85f60c56ef0dd2bee780b5ede268f6a8064ee9477f1cccca4f6: Status 404 returned error can't find the container with id 7739ccca0bbcf85f60c56ef0dd2bee780b5ede268f6a8064ee9477f1cccca4f6
Sep 09 21:35:00.615678 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.615643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c45ff568-gs674" event={"ID":"62956573-ed81-4758-9a6f-493e609e299f","Type":"ContainerStarted","Data":"4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49"}
Sep 09 21:35:00.616165 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.615704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c45ff568-gs674" event={"ID":"62956573-ed81-4758-9a6f-493e609e299f","Type":"ContainerStarted","Data":"7739ccca0bbcf85f60c56ef0dd2bee780b5ede268f6a8064ee9477f1cccca4f6"}
Sep 09 21:35:00.619104 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.617343    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b294a07-8857-420a-bfec-80269ffaf9bc" containerID="dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d" exitCode=0
Sep 09 21:35:00.619104 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.617459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd" event={"ID":"3b294a07-8857-420a-bfec-80269ffaf9bc","Type":"ContainerDied","Data":"dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d"}
Sep 09 21:35:00.619104 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.617485    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd"
Sep 09 21:35:00.619104 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.617512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc4878bb4-sfpmd" event={"ID":"3b294a07-8857-420a-bfec-80269ffaf9bc","Type":"ContainerDied","Data":"4b8e798344ae390005a3a8ae621e5828fdc95f4acdc9615c85e2f6f7b080562f"}
Sep 09 21:35:00.619104 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.617538    2563 scope.go:117] "RemoveContainer" containerID="dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d"
Sep 09 21:35:00.622070 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.621915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4" event={"ID":"c20acab5-48d1-46c0-9d06-6f03623f6987","Type":"ContainerStarted","Data":"6ec16fe34670b739a2460d7ecf287f60308639851d6ce4491890c7b640316b38"}
Sep 09 21:35:00.622070 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.621987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4" event={"ID":"c20acab5-48d1-46c0-9d06-6f03623f6987","Type":"ContainerStarted","Data":"04d6838203a35533bc32facd6d76f206cb365fb8572d82c16b3ca71820976986"}
Sep 09 21:35:00.633447 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.633063    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c45ff568-gs674" podStartSLOduration=1.633045407 podStartE2EDuration="1.633045407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:00.632145795 +0000 UTC m=+106.118028826" watchObservedRunningTime="2026-09-09 21:35:00.633045407 +0000 UTC m=+106.118928447"
Sep 09 21:35:00.645783 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.645748    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cc4878bb4-sfpmd"]
Sep 09 21:35:00.648870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.648849    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cc4878bb4-sfpmd"]
Sep 09 21:35:00.980466 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.980439    2563 scope.go:117] "RemoveContainer" containerID="dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d"
Sep 09 21:35:00.980895 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:35:00.980861    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d\": container with ID starting with dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d not found: ID does not exist" containerID="dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d"
Sep 09 21:35:00.980994 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:00.980907    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d"} err="failed to get container status \"dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d\": rpc error: code = NotFound desc = could not find container \"dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d\": container with ID starting with dd30538f6d91d5abffdeeea64a117f036ad3bc62801817b1edfd3f356af9310d not found: ID does not exist"
Sep 09 21:35:01.047179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:01.047127    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b294a07-8857-420a-bfec-80269ffaf9bc" path="/var/lib/kubelet/pods/3b294a07-8857-420a-bfec-80269ffaf9bc/volumes"
Sep 09 21:35:01.631111 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:01.631080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0fe9a7f-5caf-4e35-848e-b9d07ed13697","Type":"ContainerStarted","Data":"c85ba54e6a6b8cf7995a2b52321b5fe58f69e6c04afa67f89cb7f42cdd2104b0"}
Sep 09 21:35:01.631558 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:01.631152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0fe9a7f-5caf-4e35-848e-b9d07ed13697","Type":"ContainerStarted","Data":"9279644804d598016a9d47a25c4c86f72d78f9e9a32c74a20ae1349fd8d6bfff"}
Sep 09 21:35:02.639625 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:02.639588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0fe9a7f-5caf-4e35-848e-b9d07ed13697","Type":"ContainerStarted","Data":"51a1d6dc4216188e760da93e9da6047141e234143b6e78c3098eed77cf907371"}
Sep 09 21:35:02.640036 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:02.639681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0fe9a7f-5caf-4e35-848e-b9d07ed13697","Type":"ContainerStarted","Data":"8437592c34a1345b68bb0f1a4e9d2c21cf821c69b65ffba129e1a6496d3f584c"}
Sep 09 21:35:02.640036 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:02.639747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0fe9a7f-5caf-4e35-848e-b9d07ed13697","Type":"ContainerStarted","Data":"568212423b8ebf82c2d27891ed9e8aaa7cfa3eca97d0a1bb4dde27dc23e9e7b2"}
Sep 09 21:35:02.642379 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:02.642358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4" event={"ID":"c20acab5-48d1-46c0-9d06-6f03623f6987","Type":"ContainerStarted","Data":"11e628ed13143b645f0b346c1bf969a507760cd81c6a950d0b27aa5365a3c42c"}
Sep 09 21:35:02.642469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:02.642412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4" event={"ID":"c20acab5-48d1-46c0-9d06-6f03623f6987","Type":"ContainerStarted","Data":"b76c805a7be9c3ef2a2c39b68007316c942cd1fa656bd13bd80c22c01a3578ea"}
Sep 09 21:35:03.649536 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:03.649496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4" event={"ID":"c20acab5-48d1-46c0-9d06-6f03623f6987","Type":"ContainerStarted","Data":"12e05624f589062943d1f87faf37bc3c2b1937ac415fafcca0ca4e8b1b56a007"}
Sep 09 21:35:03.650147 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:03.649660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:35:03.671537 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:03.671484    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4" podStartSLOduration=2.818858344 podStartE2EDuration="8.671468509s" totalImagesPullingTime="5.852610165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:03.671108896 +0000 UTC m=+109.156991933" watchObservedRunningTime="2026-09-09 21:35:03.671468509 +0000 UTC m=+109.157351563"
Sep 09 21:35:04.660921 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:04.660879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0fe9a7f-5caf-4e35-848e-b9d07ed13697","Type":"ContainerStarted","Data":"e24cd040a25dfb71ed3bb2f70927002abc6b3a8c1728b9aeb2196ed859d590f9"}
Sep 09 21:35:04.666978 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:04.666953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55d7cb6886-pp7s4"
Sep 09 21:35:04.690567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:04.690513    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.173099256 podStartE2EDuration="10.690498514s" totalImagesPullingTime="6.488929389s" totalInitContainerRuntime="28.469869ms" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:04.688683407 +0000 UTC m=+110.174566444" watchObservedRunningTime="2026-09-09 21:35:04.690498514 +0000 UTC m=+110.176381556"
Sep 09 21:35:10.093319 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:10.093283    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:35:10.093881 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:10.093452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:35:10.098066 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:10.098049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:35:10.685872 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:10.685842    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:35:10.739482 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:10.739450    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6448c9c59d-8qxxp"]
Sep 09 21:35:22.710707 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:22.710637    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78d9668758-nqmgs" podUID="0bc25d7b-d1ac-444f-b3f3-8147ac964dec" containerName="console" containerID="cri-o://3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a" gracePeriod=15
Sep 09 21:35:22.951167 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:22.951146    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:35:23.071788 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.071686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-serving-cert\") pod \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") "
Sep 09 21:35:23.071927 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.071799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2fsf\" (UniqueName: \"kubernetes.io/projected/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-kube-api-access-h2fsf\") pod \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") "
Sep 09 21:35:23.071927 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.071843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-config\") pod \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") "
Sep 09 21:35:23.071927 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.071887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-service-ca\") pod \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") "
Sep 09 21:35:23.072078 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.071925    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-oauth-serving-cert\") pod \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") "
Sep 09 21:35:23.072078 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.071979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-oauth-config\") pod \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\" (UID: \"0bc25d7b-d1ac-444f-b3f3-8147ac964dec\") "
Sep 09 21:35:23.072308 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.072264    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-config" (OuterVolumeSpecName: "console-config") pod "0bc25d7b-d1ac-444f-b3f3-8147ac964dec" (UID: "0bc25d7b-d1ac-444f-b3f3-8147ac964dec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:23.072308 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.072281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0bc25d7b-d1ac-444f-b3f3-8147ac964dec" (UID: "0bc25d7b-d1ac-444f-b3f3-8147ac964dec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:23.072659 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.072312    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-service-ca" (OuterVolumeSpecName: "service-ca") pod "0bc25d7b-d1ac-444f-b3f3-8147ac964dec" (UID: "0bc25d7b-d1ac-444f-b3f3-8147ac964dec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:23.074213 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.074189    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0bc25d7b-d1ac-444f-b3f3-8147ac964dec" (UID: "0bc25d7b-d1ac-444f-b3f3-8147ac964dec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:35:23.074306 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.074221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-kube-api-access-h2fsf" (OuterVolumeSpecName: "kube-api-access-h2fsf") pod "0bc25d7b-d1ac-444f-b3f3-8147ac964dec" (UID: "0bc25d7b-d1ac-444f-b3f3-8147ac964dec"). InnerVolumeSpecName "kube-api-access-h2fsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:35:23.074306 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.074242    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0bc25d7b-d1ac-444f-b3f3-8147ac964dec" (UID: "0bc25d7b-d1ac-444f-b3f3-8147ac964dec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:35:23.172613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.172572    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2fsf\" (UniqueName: \"kubernetes.io/projected/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-kube-api-access-h2fsf\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:23.172613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.172606    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:23.172613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.172615    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:23.172613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.172623    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-oauth-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:23.172909 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.172632    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-oauth-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:23.172909 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.172640    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc25d7b-d1ac-444f-b3f3-8147ac964dec-console-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:23.731795 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.731761    2563 generic.go:356] "Generic (PLEG): container finished" podID="0bc25d7b-d1ac-444f-b3f3-8147ac964dec" containerID="3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a" exitCode=0
Sep 09 21:35:23.732227 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.731809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d9668758-nqmgs"
Sep 09 21:35:23.732227 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.731841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d9668758-nqmgs" event={"ID":"0bc25d7b-d1ac-444f-b3f3-8147ac964dec","Type":"ContainerDied","Data":"3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a"}
Sep 09 21:35:23.732227 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.731885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d9668758-nqmgs" event={"ID":"0bc25d7b-d1ac-444f-b3f3-8147ac964dec","Type":"ContainerDied","Data":"9af502e21157b816679f81b6e170c2abcf80a6d39e3d22d95499da20d6ec6b15"}
Sep 09 21:35:23.732227 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.731904    2563 scope.go:117] "RemoveContainer" containerID="3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a"
Sep 09 21:35:23.740683 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.740665    2563 scope.go:117] "RemoveContainer" containerID="3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a"
Sep 09 21:35:23.741011 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:35:23.740993    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a\": container with ID starting with 3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a not found: ID does not exist" containerID="3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a"
Sep 09 21:35:23.741069 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.741019    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a"} err="failed to get container status \"3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a\": rpc error: code = NotFound desc = could not find container \"3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a\": container with ID starting with 3f0df5afb3d9ac4bd8819918832c93db5cbc9803491d1d941448ab5c5c08bb9a not found: ID does not exist"
Sep 09 21:35:23.755227 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.755198    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d9668758-nqmgs"]
Sep 09 21:35:23.758940 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:23.758917    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78d9668758-nqmgs"]
Sep 09 21:35:24.889277 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:24.889243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:35:24.891695 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:24.891662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ae5a6d6-2f5b-4af6-ab7a-b629addbf014-metrics-certs\") pod \"network-metrics-daemon-h2p64\" (UID: \"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014\") " pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:35:25.045309 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:25.045264    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc25d7b-d1ac-444f-b3f3-8147ac964dec" path="/var/lib/kubelet/pods/0bc25d7b-d1ac-444f-b3f3-8147ac964dec/volumes"
Sep 09 21:35:25.161923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:25.161845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5t55x\""
Sep 09 21:35:25.169964 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:25.169943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2p64"
Sep 09 21:35:25.466351 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:25.466257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h2p64"]
Sep 09 21:35:25.469128 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:35:25.469099    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae5a6d6_2f5b_4af6_ab7a_b629addbf014.slice/crio-b033b1544aa219c953877218d8b61a7cc46bee96e227c371ce11cce51b67d536 WatchSource:0}: Error finding container b033b1544aa219c953877218d8b61a7cc46bee96e227c371ce11cce51b67d536: Status 404 returned error can't find the container with id b033b1544aa219c953877218d8b61a7cc46bee96e227c371ce11cce51b67d536
Sep 09 21:35:25.742012 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:25.741914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h2p64" event={"ID":"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014","Type":"ContainerStarted","Data":"b033b1544aa219c953877218d8b61a7cc46bee96e227c371ce11cce51b67d536"}
Sep 09 21:35:27.753532 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:27.753433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h2p64" event={"ID":"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014","Type":"ContainerStarted","Data":"3775b609f1bb1678abadda0e13a2f2782c50ed89dd22ec287399c373c4aea6a6"}
Sep 09 21:35:27.753532 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:27.753496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h2p64" event={"ID":"0ae5a6d6-2f5b-4af6-ab7a-b629addbf014","Type":"ContainerStarted","Data":"d1f2f7ac0d6f4ed655db657f079ed089ebec195acbf10b9638c60a332107fc69"}
Sep 09 21:35:27.769783 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:27.769677    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h2p64" podStartSLOduration=131.083428107 podStartE2EDuration="2m12.769660216s" totalImagesPullingTime="1.686232109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:27.769012863 +0000 UTC m=+133.254895906" watchObservedRunningTime="2026-09-09 21:35:27.769660216 +0000 UTC m=+133.255543292"
Sep 09 21:35:35.760300 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:35.760249    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6448c9c59d-8qxxp" podUID="647c4f0a-1eea-4a84-8900-47b5cf7f88d2" containerName="console" containerID="cri-o://46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c" gracePeriod=15
Sep 09 21:35:36.035349 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.035266    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:35:36.192414 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-oauth-serving-cert\") pod \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") "
Sep 09 21:35:36.192414 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-serving-cert\") pod \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") "
Sep 09 21:35:36.192660 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-oauth-config\") pod \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") "
Sep 09 21:35:36.192660 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-config\") pod \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") "
Sep 09 21:35:36.192660 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-trusted-ca-bundle\") pod \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") "
Sep 09 21:35:36.192660 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2cpj\" (UniqueName: \"kubernetes.io/projected/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-kube-api-access-h2cpj\") pod \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") "
Sep 09 21:35:36.192892 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-service-ca\") pod \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\" (UID: \"647c4f0a-1eea-4a84-8900-47b5cf7f88d2\") "
Sep 09 21:35:36.192892 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "647c4f0a-1eea-4a84-8900-47b5cf7f88d2" (UID: "647c4f0a-1eea-4a84-8900-47b5cf7f88d2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:36.193006 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.192972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-config" (OuterVolumeSpecName: "console-config") pod "647c4f0a-1eea-4a84-8900-47b5cf7f88d2" (UID: "647c4f0a-1eea-4a84-8900-47b5cf7f88d2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:36.193057 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.193033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "647c4f0a-1eea-4a84-8900-47b5cf7f88d2" (UID: "647c4f0a-1eea-4a84-8900-47b5cf7f88d2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:36.193136 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.193116    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-oauth-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:36.193186 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.193138    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:36.193186 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.193151    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-trusted-ca-bundle\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:36.193186 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.193152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-service-ca" (OuterVolumeSpecName: "service-ca") pod "647c4f0a-1eea-4a84-8900-47b5cf7f88d2" (UID: "647c4f0a-1eea-4a84-8900-47b5cf7f88d2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:36.194917 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.194883    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-kube-api-access-h2cpj" (OuterVolumeSpecName: "kube-api-access-h2cpj") pod "647c4f0a-1eea-4a84-8900-47b5cf7f88d2" (UID: "647c4f0a-1eea-4a84-8900-47b5cf7f88d2"). InnerVolumeSpecName "kube-api-access-h2cpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:35:36.195249 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.195222    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "647c4f0a-1eea-4a84-8900-47b5cf7f88d2" (UID: "647c4f0a-1eea-4a84-8900-47b5cf7f88d2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:35:36.195249 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.195238    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "647c4f0a-1eea-4a84-8900-47b5cf7f88d2" (UID: "647c4f0a-1eea-4a84-8900-47b5cf7f88d2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:35:36.293916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.293831    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2cpj\" (UniqueName: \"kubernetes.io/projected/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-kube-api-access-h2cpj\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:36.293916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.293867    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:36.293916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.293877    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:36.293916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.293885    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/647c4f0a-1eea-4a84-8900-47b5cf7f88d2-console-oauth-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:35:36.792119 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.792084    2563 generic.go:356] "Generic (PLEG): container finished" podID="647c4f0a-1eea-4a84-8900-47b5cf7f88d2" containerID="46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c" exitCode=0
Sep 09 21:35:36.792563 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.792132    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6448c9c59d-8qxxp"
Sep 09 21:35:36.792563 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.792172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6448c9c59d-8qxxp" event={"ID":"647c4f0a-1eea-4a84-8900-47b5cf7f88d2","Type":"ContainerDied","Data":"46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c"}
Sep 09 21:35:36.792563 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.792224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6448c9c59d-8qxxp" event={"ID":"647c4f0a-1eea-4a84-8900-47b5cf7f88d2","Type":"ContainerDied","Data":"7ef3536cd8e038b562e10f2eaeaa8be1910072b093efecd78f98a9b61b34e505"}
Sep 09 21:35:36.792563 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.792245    2563 scope.go:117] "RemoveContainer" containerID="46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c"
Sep 09 21:35:36.806234 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.806215    2563 scope.go:117] "RemoveContainer" containerID="46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c"
Sep 09 21:35:36.806542 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:35:36.806524    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c\": container with ID starting with 46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c not found: ID does not exist" containerID="46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c"
Sep 09 21:35:36.806587 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.806551    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c"} err="failed to get container status \"46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c\": rpc error: code = NotFound desc = could not find container \"46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c\": container with ID starting with 46f53af00d14fe6095a9d69953bd02baa85aee4afac0e6dd49153e6b0ed7ed8c not found: ID does not exist"
Sep 09 21:35:36.816276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.816245    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6448c9c59d-8qxxp"]
Sep 09 21:35:36.819432 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:36.819410    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6448c9c59d-8qxxp"]
Sep 09 21:35:37.045456 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:35:37.045371    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647c4f0a-1eea-4a84-8900-47b5cf7f88d2" path="/var/lib/kubelet/pods/647c4f0a-1eea-4a84-8900-47b5cf7f88d2/volumes"
Sep 09 21:36:17.510062 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510026    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"]
Sep 09 21:36:17.510500 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510390    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc25d7b-d1ac-444f-b3f3-8147ac964dec" containerName="console"
Sep 09 21:36:17.510500 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510411    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc25d7b-d1ac-444f-b3f3-8147ac964dec" containerName="console"
Sep 09 21:36:17.510500 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510440    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b294a07-8857-420a-bfec-80269ffaf9bc" containerName="registry"
Sep 09 21:36:17.510500 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510457    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b294a07-8857-420a-bfec-80269ffaf9bc" containerName="registry"
Sep 09 21:36:17.510500 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510479    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647c4f0a-1eea-4a84-8900-47b5cf7f88d2" containerName="console"
Sep 09 21:36:17.510500 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510488    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="647c4f0a-1eea-4a84-8900-47b5cf7f88d2" containerName="console"
Sep 09 21:36:17.510705 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510557    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="647c4f0a-1eea-4a84-8900-47b5cf7f88d2" containerName="console"
Sep 09 21:36:17.510705 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510569    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc25d7b-d1ac-444f-b3f3-8147ac964dec" containerName="console"
Sep 09 21:36:17.510705 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.510586    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b294a07-8857-420a-bfec-80269ffaf9bc" containerName="registry"
Sep 09 21:36:17.513126 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.513112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.516698 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.516680    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 21:36:17.516857 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.516711    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 21:36:17.517039 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.517020    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4np7v\""
Sep 09 21:36:17.517095 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.517037    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 21:36:17.517095 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.517051    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 21:36:17.517283 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.517269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 21:36:17.521772 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.521753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 21:36:17.526465 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.526442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"]
Sep 09 21:36:17.603696 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.603659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpdk\" (UniqueName: \"kubernetes.io/projected/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-kube-api-access-zvpdk\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.603696 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.603693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-serving-certs-ca-bundle\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.603938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.603747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.603938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.603779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-secret-telemeter-client\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.603938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.603808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-federate-client-tls\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.603938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.603828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.603938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.603864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-metrics-client-ca\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.603938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.603882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-telemeter-client-tls\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.705185 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.705150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.705325 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.705188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-secret-telemeter-client\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.705325 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.705211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-federate-client-tls\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.705325 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.705234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.705325 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.705280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-metrics-client-ca\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.705514 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.705401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-telemeter-client-tls\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.705514 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.705468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpdk\" (UniqueName: \"kubernetes.io/projected/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-kube-api-access-zvpdk\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.705514 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.705495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-serving-certs-ca-bundle\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.706081 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.706057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-metrics-client-ca\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.706195 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.706174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-serving-certs-ca-bundle\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.706235 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.706218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.708064 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.708040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-secret-telemeter-client\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.708323 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.708300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.708407 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.708351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-federate-client-tls\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.708469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.708442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-telemeter-client-tls\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.714685 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.714662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpdk\" (UniqueName: \"kubernetes.io/projected/18c315f3-b0e1-42a9-878f-db49d6a0ffd9-kube-api-access-zvpdk\") pod \"telemeter-client-86b69f4dc7-pj7rk\" (UID: \"18c315f3-b0e1-42a9-878f-db49d6a0ffd9\") " pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:17.822388 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:17.822291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"
Sep 09 21:36:18.101423 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:18.101388    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk"]
Sep 09 21:36:18.102740 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:36:18.102697    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c315f3_b0e1_42a9_878f_db49d6a0ffd9.slice/crio-183f7e58cb7fd36655a9e2dd13c415a11375dc0f2e9f0433a497de4124dcfd74 WatchSource:0}: Error finding container 183f7e58cb7fd36655a9e2dd13c415a11375dc0f2e9f0433a497de4124dcfd74: Status 404 returned error can't find the container with id 183f7e58cb7fd36655a9e2dd13c415a11375dc0f2e9f0433a497de4124dcfd74
Sep 09 21:36:18.959348 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:18.959304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk" event={"ID":"18c315f3-b0e1-42a9-878f-db49d6a0ffd9","Type":"ContainerStarted","Data":"183f7e58cb7fd36655a9e2dd13c415a11375dc0f2e9f0433a497de4124dcfd74"}
Sep 09 21:36:20.969375 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:20.969340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk" event={"ID":"18c315f3-b0e1-42a9-878f-db49d6a0ffd9","Type":"ContainerStarted","Data":"9763c9eceb9d02d7b6ddc0b264f28254c6a914ecb57ec21be04c9adbc5c454fc"}
Sep 09 21:36:20.969809 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:20.969399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk" event={"ID":"18c315f3-b0e1-42a9-878f-db49d6a0ffd9","Type":"ContainerStarted","Data":"6e210ab0fe2ac8ce3b83240b172c023eaa9b3f4dca2408c320ad6af5004bd04b"}
Sep 09 21:36:20.969809 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:20.969420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk" event={"ID":"18c315f3-b0e1-42a9-878f-db49d6a0ffd9","Type":"ContainerStarted","Data":"e7e32e8d254f0307611881f2e90c208c03cfced8b07544e48d513751a4b839c2"}
Sep 09 21:36:20.992174 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:20.992124    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-86b69f4dc7-pj7rk" podStartSLOduration=1.553152546 podStartE2EDuration="3.992107329s" totalImagesPullingTime="2.438954783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:36:20.989415901 +0000 UTC m=+186.475298938" watchObservedRunningTime="2026-09-09 21:36:20.992107329 +0000 UTC m=+186.477990370"
Sep 09 21:36:21.068848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.068811    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57f5fcd8b4-qlmgd"]
Sep 09 21:36:21.071065 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.071042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.085823 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.085794    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f5fcd8b4-qlmgd"]
Sep 09 21:36:21.136569 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.136537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-service-ca\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.136815 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.136588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwd6x\" (UniqueName: \"kubernetes.io/projected/b2689d5e-199b-4d18-92a1-1d5257ecb58b-kube-api-access-pwd6x\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.136815 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.136666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-serving-cert\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.136815 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.136690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-oauth-serving-cert\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.136815 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.136772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-config\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.136815 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.136794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-trusted-ca-bundle\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.136999 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.136857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-oauth-config\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.237940 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.237853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-serving-cert\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.237940 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.237892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-oauth-serving-cert\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.237940 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.237938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-config\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.238183 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.237965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-trusted-ca-bundle\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.238183 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.238007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-oauth-config\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.238183 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.238151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-service-ca\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.238330 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.238231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwd6x\" (UniqueName: \"kubernetes.io/projected/b2689d5e-199b-4d18-92a1-1d5257ecb58b-kube-api-access-pwd6x\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.238796 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.238769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-oauth-serving-cert\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.238875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.238817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-config\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.238875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.238846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-service-ca\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.238954 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.238885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-trusted-ca-bundle\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.241074 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.241038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-serving-cert\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.241074 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.241056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-oauth-config\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.247008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.246984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwd6x\" (UniqueName: \"kubernetes.io/projected/b2689d5e-199b-4d18-92a1-1d5257ecb58b-kube-api-access-pwd6x\") pod \"console-57f5fcd8b4-qlmgd\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") " pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.380211 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.380173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:21.653289 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.653155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f5fcd8b4-qlmgd"]
Sep 09 21:36:21.655680 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:36:21.655652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2689d5e_199b_4d18_92a1_1d5257ecb58b.slice/crio-11560245caff4a7db87a4da45bc737e41b7f72824fd01335cb555d3ad14ee879 WatchSource:0}: Error finding container 11560245caff4a7db87a4da45bc737e41b7f72824fd01335cb555d3ad14ee879: Status 404 returned error can't find the container with id 11560245caff4a7db87a4da45bc737e41b7f72824fd01335cb555d3ad14ee879
Sep 09 21:36:21.974989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.974892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5fcd8b4-qlmgd" event={"ID":"b2689d5e-199b-4d18-92a1-1d5257ecb58b","Type":"ContainerStarted","Data":"a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7"}
Sep 09 21:36:21.974989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.974947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5fcd8b4-qlmgd" event={"ID":"b2689d5e-199b-4d18-92a1-1d5257ecb58b","Type":"ContainerStarted","Data":"11560245caff4a7db87a4da45bc737e41b7f72824fd01335cb555d3ad14ee879"}
Sep 09 21:36:21.992934 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:21.992861    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57f5fcd8b4-qlmgd" podStartSLOduration=0.992841456 podStartE2EDuration="992.841456ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:36:21.99022842 +0000 UTC m=+187.476111476" watchObservedRunningTime="2026-09-09 21:36:21.992841456 +0000 UTC m=+187.478724494"
Sep 09 21:36:31.380536 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:31.380494    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:31.380976 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:31.380553    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:31.385056 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:31.385036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:32.017912 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.017884    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:36:32.059752 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.059705    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c45ff568-gs674"]
Sep 09 21:36:32.128016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.127987    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-97ffcbf56-gs4lf"]
Sep 09 21:36:32.130234 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.130216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.141772 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.141745    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-97ffcbf56-gs4lf"]
Sep 09 21:36:32.237601 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.237553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlz5c\" (UniqueName: \"kubernetes.io/projected/292b56a0-6b1b-45b6-aa9d-6528ab91923b-kube-api-access-nlz5c\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.237601 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.237599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-oauth-config\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.237837 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.237623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-config\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.237837 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.237646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-oauth-serving-cert\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.237837 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.237761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-serving-cert\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.237837 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.237810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-trusted-ca-bundle\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.237964 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.237862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-service-ca\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339015 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.338921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-serving-cert\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339015 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.338971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-trusted-ca-bundle\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339015 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.339008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-service-ca\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339236 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.339055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlz5c\" (UniqueName: \"kubernetes.io/projected/292b56a0-6b1b-45b6-aa9d-6528ab91923b-kube-api-access-nlz5c\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339236 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.339085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-oauth-config\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339236 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.339134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-config\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339236 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.339157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-oauth-serving-cert\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339852 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.339822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-service-ca\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.339987 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.339950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-oauth-serving-cert\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.340029 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.340008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-trusted-ca-bundle\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.340193 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.340169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-config\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.341671 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.341641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-oauth-config\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.341802 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.341747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-serving-cert\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.347440 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.347415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlz5c\" (UniqueName: \"kubernetes.io/projected/292b56a0-6b1b-45b6-aa9d-6528ab91923b-kube-api-access-nlz5c\") pod \"console-97ffcbf56-gs4lf\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") " pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.439179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.439140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:32.706341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:32.706164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-97ffcbf56-gs4lf"]
Sep 09 21:36:32.709174 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:36:32.709149    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292b56a0_6b1b_45b6_aa9d_6528ab91923b.slice/crio-503c9fa12db0c6dc69b8a8bc67e174d636ebc059924e1426368883af8582c95e WatchSource:0}: Error finding container 503c9fa12db0c6dc69b8a8bc67e174d636ebc059924e1426368883af8582c95e: Status 404 returned error can't find the container with id 503c9fa12db0c6dc69b8a8bc67e174d636ebc059924e1426368883af8582c95e
Sep 09 21:36:33.019421 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:33.019323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97ffcbf56-gs4lf" event={"ID":"292b56a0-6b1b-45b6-aa9d-6528ab91923b","Type":"ContainerStarted","Data":"78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad"}
Sep 09 21:36:33.019421 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:33.019385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97ffcbf56-gs4lf" event={"ID":"292b56a0-6b1b-45b6-aa9d-6528ab91923b","Type":"ContainerStarted","Data":"503c9fa12db0c6dc69b8a8bc67e174d636ebc059924e1426368883af8582c95e"}
Sep 09 21:36:33.038521 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:33.038469    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-97ffcbf56-gs4lf" podStartSLOduration=1.038452365 podStartE2EDuration="1.038452365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:36:33.037254517 +0000 UTC m=+198.523137571" watchObservedRunningTime="2026-09-09 21:36:33.038452365 +0000 UTC m=+198.524335402"
Sep 09 21:36:42.439595 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:42.439549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:42.440071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:42.439622    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:42.444303 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:42.444281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:43.063737 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:43.063685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:36:43.102920 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:43.102887    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f5fcd8b4-qlmgd"]
Sep 09 21:36:57.081214 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.081167    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c45ff568-gs674" podUID="62956573-ed81-4758-9a6f-493e609e299f" containerName="console" containerID="cri-o://4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49" gracePeriod=15
Sep 09 21:36:57.316065 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.316040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:36:57.345984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.345902    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-oauth-config\") pod \"62956573-ed81-4758-9a6f-493e609e299f\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") "
Sep 09 21:36:57.345984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.345952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-trusted-ca-bundle\") pod \"62956573-ed81-4758-9a6f-493e609e299f\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") "
Sep 09 21:36:57.345984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.345977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-serving-cert\") pod \"62956573-ed81-4758-9a6f-493e609e299f\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") "
Sep 09 21:36:57.346288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.346007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-console-config\") pod \"62956573-ed81-4758-9a6f-493e609e299f\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") "
Sep 09 21:36:57.346288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.346030    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-oauth-serving-cert\") pod \"62956573-ed81-4758-9a6f-493e609e299f\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") "
Sep 09 21:36:57.346288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.346100    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmpzl\" (UniqueName: \"kubernetes.io/projected/62956573-ed81-4758-9a6f-493e609e299f-kube-api-access-mmpzl\") pod \"62956573-ed81-4758-9a6f-493e609e299f\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") "
Sep 09 21:36:57.346288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.346256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-service-ca\") pod \"62956573-ed81-4758-9a6f-493e609e299f\" (UID: \"62956573-ed81-4758-9a6f-493e609e299f\") "
Sep 09 21:36:57.346848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.346599    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "62956573-ed81-4758-9a6f-493e609e299f" (UID: "62956573-ed81-4758-9a6f-493e609e299f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:36:57.346967 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.346651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-console-config" (OuterVolumeSpecName: "console-config") pod "62956573-ed81-4758-9a6f-493e609e299f" (UID: "62956573-ed81-4758-9a6f-493e609e299f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:36:57.346967 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.346776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "62956573-ed81-4758-9a6f-493e609e299f" (UID: "62956573-ed81-4758-9a6f-493e609e299f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:36:57.346967 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.346809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-service-ca" (OuterVolumeSpecName: "service-ca") pod "62956573-ed81-4758-9a6f-493e609e299f" (UID: "62956573-ed81-4758-9a6f-493e609e299f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:36:57.348553 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.348527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "62956573-ed81-4758-9a6f-493e609e299f" (UID: "62956573-ed81-4758-9a6f-493e609e299f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:36:57.348643 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.348560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "62956573-ed81-4758-9a6f-493e609e299f" (UID: "62956573-ed81-4758-9a6f-493e609e299f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:36:57.348643 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.348591    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62956573-ed81-4758-9a6f-493e609e299f-kube-api-access-mmpzl" (OuterVolumeSpecName: "kube-api-access-mmpzl") pod "62956573-ed81-4758-9a6f-493e609e299f" (UID: "62956573-ed81-4758-9a6f-493e609e299f"). InnerVolumeSpecName "kube-api-access-mmpzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:36:57.447702 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.447665    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmpzl\" (UniqueName: \"kubernetes.io/projected/62956573-ed81-4758-9a6f-493e609e299f-kube-api-access-mmpzl\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.447702 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.447693    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.447702 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.447704    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-oauth-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.447702 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.447712    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-trusted-ca-bundle\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.448030 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.447739    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62956573-ed81-4758-9a6f-493e609e299f-console-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.448030 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.447747    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-console-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.448030 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:57.447755    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62956573-ed81-4758-9a6f-493e609e299f-oauth-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:36:58.117558 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.117523    2563 generic.go:356] "Generic (PLEG): container finished" podID="62956573-ed81-4758-9a6f-493e609e299f" containerID="4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49" exitCode=0
Sep 09 21:36:58.118016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.117574    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c45ff568-gs674"
Sep 09 21:36:58.118016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.117587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c45ff568-gs674" event={"ID":"62956573-ed81-4758-9a6f-493e609e299f","Type":"ContainerDied","Data":"4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49"}
Sep 09 21:36:58.118016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.117625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c45ff568-gs674" event={"ID":"62956573-ed81-4758-9a6f-493e609e299f","Type":"ContainerDied","Data":"7739ccca0bbcf85f60c56ef0dd2bee780b5ede268f6a8064ee9477f1cccca4f6"}
Sep 09 21:36:58.118016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.117644    2563 scope.go:117] "RemoveContainer" containerID="4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49"
Sep 09 21:36:58.126238 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.126221    2563 scope.go:117] "RemoveContainer" containerID="4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49"
Sep 09 21:36:58.126535 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:36:58.126514    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49\": container with ID starting with 4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49 not found: ID does not exist" containerID="4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49"
Sep 09 21:36:58.126577 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.126546    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49"} err="failed to get container status \"4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49\": rpc error: code = NotFound desc = could not find container \"4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49\": container with ID starting with 4d3aa51da302cc09226bbed3d740f574892a64fbebc89a5f831c852f9ac71b49 not found: ID does not exist"
Sep 09 21:36:58.137394 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.137371    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c45ff568-gs674"]
Sep 09 21:36:58.140534 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:58.140511    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c45ff568-gs674"]
Sep 09 21:36:59.045435 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:36:59.045392    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62956573-ed81-4758-9a6f-493e609e299f" path="/var/lib/kubelet/pods/62956573-ed81-4758-9a6f-493e609e299f/volumes"
Sep 09 21:37:08.129359 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.129315    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57f5fcd8b4-qlmgd" podUID="b2689d5e-199b-4d18-92a1-1d5257ecb58b" containerName="console" containerID="cri-o://a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7" gracePeriod=15
Sep 09 21:37:08.376199 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.376176    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:37:08.428470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428369    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-oauth-config\") pod \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") "
Sep 09 21:37:08.428470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-serving-cert\") pod \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") "
Sep 09 21:37:08.428470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-oauth-serving-cert\") pod \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") "
Sep 09 21:37:08.428853 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428482    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-config\") pod \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") "
Sep 09 21:37:08.428853 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwd6x\" (UniqueName: \"kubernetes.io/projected/b2689d5e-199b-4d18-92a1-1d5257ecb58b-kube-api-access-pwd6x\") pod \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") "
Sep 09 21:37:08.428853 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-trusted-ca-bundle\") pod \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") "
Sep 09 21:37:08.428853 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428553    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-service-ca\") pod \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\" (UID: \"b2689d5e-199b-4d18-92a1-1d5257ecb58b\") "
Sep 09 21:37:08.429019 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428958    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b2689d5e-199b-4d18-92a1-1d5257ecb58b" (UID: "b2689d5e-199b-4d18-92a1-1d5257ecb58b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:37:08.429019 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.428985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b2689d5e-199b-4d18-92a1-1d5257ecb58b" (UID: "b2689d5e-199b-4d18-92a1-1d5257ecb58b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:37:08.429124 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.429033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-service-ca" (OuterVolumeSpecName: "service-ca") pod "b2689d5e-199b-4d18-92a1-1d5257ecb58b" (UID: "b2689d5e-199b-4d18-92a1-1d5257ecb58b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:37:08.429322 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.429240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-config" (OuterVolumeSpecName: "console-config") pod "b2689d5e-199b-4d18-92a1-1d5257ecb58b" (UID: "b2689d5e-199b-4d18-92a1-1d5257ecb58b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:37:08.430877 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.430852    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2689d5e-199b-4d18-92a1-1d5257ecb58b-kube-api-access-pwd6x" (OuterVolumeSpecName: "kube-api-access-pwd6x") pod "b2689d5e-199b-4d18-92a1-1d5257ecb58b" (UID: "b2689d5e-199b-4d18-92a1-1d5257ecb58b"). InnerVolumeSpecName "kube-api-access-pwd6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:37:08.430877 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.430859    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b2689d5e-199b-4d18-92a1-1d5257ecb58b" (UID: "b2689d5e-199b-4d18-92a1-1d5257ecb58b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:37:08.430877 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.430873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b2689d5e-199b-4d18-92a1-1d5257ecb58b" (UID: "b2689d5e-199b-4d18-92a1-1d5257ecb58b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:37:08.530026 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.529986    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-oauth-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:37:08.530026 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.530020    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:37:08.530026 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.530029    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-oauth-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:37:08.530265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.530038    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-console-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:37:08.530265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.530048    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwd6x\" (UniqueName: \"kubernetes.io/projected/b2689d5e-199b-4d18-92a1-1d5257ecb58b-kube-api-access-pwd6x\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:37:08.530265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.530058    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-trusted-ca-bundle\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:37:08.530265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:08.530066    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2689d5e-199b-4d18-92a1-1d5257ecb58b-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:37:09.163562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.163527    2563 generic.go:356] "Generic (PLEG): container finished" podID="b2689d5e-199b-4d18-92a1-1d5257ecb58b" containerID="a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7" exitCode=0
Sep 09 21:37:09.164047 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.163574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5fcd8b4-qlmgd" event={"ID":"b2689d5e-199b-4d18-92a1-1d5257ecb58b","Type":"ContainerDied","Data":"a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7"}
Sep 09 21:37:09.164047 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.163581    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5fcd8b4-qlmgd"
Sep 09 21:37:09.164047 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.163615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5fcd8b4-qlmgd" event={"ID":"b2689d5e-199b-4d18-92a1-1d5257ecb58b","Type":"ContainerDied","Data":"11560245caff4a7db87a4da45bc737e41b7f72824fd01335cb555d3ad14ee879"}
Sep 09 21:37:09.164047 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.163635    2563 scope.go:117] "RemoveContainer" containerID="a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7"
Sep 09 21:37:09.171820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.171802    2563 scope.go:117] "RemoveContainer" containerID="a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7"
Sep 09 21:37:09.172106 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:37:09.172079    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7\": container with ID starting with a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7 not found: ID does not exist" containerID="a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7"
Sep 09 21:37:09.172159 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.172115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7"} err="failed to get container status \"a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7\": rpc error: code = NotFound desc = could not find container \"a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7\": container with ID starting with a91aaaf95669a826a12d343f1a74cf292d69da2bce2bacccc59e9f6c7d5f3ed7 not found: ID does not exist"
Sep 09 21:37:09.179516 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.179466    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f5fcd8b4-qlmgd"]
Sep 09 21:37:09.181529 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:09.181507    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57f5fcd8b4-qlmgd"]
Sep 09 21:37:11.045155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:11.045125    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2689d5e-199b-4d18-92a1-1d5257ecb58b" path="/var/lib/kubelet/pods/b2689d5e-199b-4d18-92a1-1d5257ecb58b/volumes"
Sep 09 21:37:48.170432 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.170387    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xb6jw"]
Sep 09 21:37:48.171058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.170873    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2689d5e-199b-4d18-92a1-1d5257ecb58b" containerName="console"
Sep 09 21:37:48.171058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.170892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2689d5e-199b-4d18-92a1-1d5257ecb58b" containerName="console"
Sep 09 21:37:48.171058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.170907    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62956573-ed81-4758-9a6f-493e609e299f" containerName="console"
Sep 09 21:37:48.171058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.170915    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62956573-ed81-4758-9a6f-493e609e299f" containerName="console"
Sep 09 21:37:48.171058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.170992    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62956573-ed81-4758-9a6f-493e609e299f" containerName="console"
Sep 09 21:37:48.171058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.171010    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2689d5e-199b-4d18-92a1-1d5257ecb58b" containerName="console"
Sep 09 21:37:48.173691 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.173671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:48.176579 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.176557    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 21:37:48.176843 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.176828    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 21:37:48.177932 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.177915    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fbbsb\""
Sep 09 21:37:48.188870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.188848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xb6jw"]
Sep 09 21:37:48.356988 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.356951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19fcd44f-a3e9-4e78-81c3-f798f302fd54-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xb6jw\" (UID: \"19fcd44f-a3e9-4e78-81c3-f798f302fd54\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:48.356988 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.356983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zttw\" (UniqueName: \"kubernetes.io/projected/19fcd44f-a3e9-4e78-81c3-f798f302fd54-kube-api-access-9zttw\") pod \"cert-manager-webhook-6486ffd997-xb6jw\" (UID: \"19fcd44f-a3e9-4e78-81c3-f798f302fd54\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:48.457851 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.457758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19fcd44f-a3e9-4e78-81c3-f798f302fd54-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xb6jw\" (UID: \"19fcd44f-a3e9-4e78-81c3-f798f302fd54\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:48.457851 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.457795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zttw\" (UniqueName: \"kubernetes.io/projected/19fcd44f-a3e9-4e78-81c3-f798f302fd54-kube-api-access-9zttw\") pod \"cert-manager-webhook-6486ffd997-xb6jw\" (UID: \"19fcd44f-a3e9-4e78-81c3-f798f302fd54\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:48.467027 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.466990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19fcd44f-a3e9-4e78-81c3-f798f302fd54-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xb6jw\" (UID: \"19fcd44f-a3e9-4e78-81c3-f798f302fd54\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:48.467276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.467255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zttw\" (UniqueName: \"kubernetes.io/projected/19fcd44f-a3e9-4e78-81c3-f798f302fd54-kube-api-access-9zttw\") pod \"cert-manager-webhook-6486ffd997-xb6jw\" (UID: \"19fcd44f-a3e9-4e78-81c3-f798f302fd54\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:48.482914 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.482886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:48.752619 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:48.752446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xb6jw"]
Sep 09 21:37:48.755204 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:37:48.755178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19fcd44f_a3e9_4e78_81c3_f798f302fd54.slice/crio-a58ac2c48227b95d20927ec92b18ca7d0bbafccc466b6e17dc019419611aed67 WatchSource:0}: Error finding container a58ac2c48227b95d20927ec92b18ca7d0bbafccc466b6e17dc019419611aed67: Status 404 returned error can't find the container with id a58ac2c48227b95d20927ec92b18ca7d0bbafccc466b6e17dc019419611aed67
Sep 09 21:37:49.315187 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:49.315150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw" event={"ID":"19fcd44f-a3e9-4e78-81c3-f798f302fd54","Type":"ContainerStarted","Data":"a58ac2c48227b95d20927ec92b18ca7d0bbafccc466b6e17dc019419611aed67"}
Sep 09 21:37:52.330280 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:52.330177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw" event={"ID":"19fcd44f-a3e9-4e78-81c3-f798f302fd54","Type":"ContainerStarted","Data":"381d9a2eb14e4a81d2e1c606806bfccb6f03e25f522f78dd957a18be1fc049a3"}
Sep 09 21:37:52.330280 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:52.330243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:52.349033 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:52.348979    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw" podStartSLOduration=1.2287854249999999 podStartE2EDuration="4.3489651s" totalImagesPullingTime="3.120179675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:37:52.347354443 +0000 UTC m=+277.833237481" watchObservedRunningTime="2026-09-09 21:37:52.3489651 +0000 UTC m=+277.834848134"
Sep 09 21:37:58.336867 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.336831    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xb6jw"
Sep 09 21:37:58.640140 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.640101    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"]
Sep 09 21:37:58.643405 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.643386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"
Sep 09 21:37:58.646084 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.646063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 21:37:58.646213 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.646111    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:37:58.646213 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.646141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wsgvt\""
Sep 09 21:37:58.651278 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.651256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"]
Sep 09 21:37:58.743799 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.743752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gg45\" (UniqueName: \"kubernetes.io/projected/3a3599c3-0b33-4364-b635-5ab62887d0ac-kube-api-access-4gg45\") pod \"openshift-lws-operator-fd8ccff4c-4ff5n\" (UID: \"3a3599c3-0b33-4364-b635-5ab62887d0ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"
Sep 09 21:37:58.743980 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.743867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a3599c3-0b33-4364-b635-5ab62887d0ac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4ff5n\" (UID: \"3a3599c3-0b33-4364-b635-5ab62887d0ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"
Sep 09 21:37:58.845066 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.845024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a3599c3-0b33-4364-b635-5ab62887d0ac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4ff5n\" (UID: \"3a3599c3-0b33-4364-b635-5ab62887d0ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"
Sep 09 21:37:58.845276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.845087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gg45\" (UniqueName: \"kubernetes.io/projected/3a3599c3-0b33-4364-b635-5ab62887d0ac-kube-api-access-4gg45\") pod \"openshift-lws-operator-fd8ccff4c-4ff5n\" (UID: \"3a3599c3-0b33-4364-b635-5ab62887d0ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"
Sep 09 21:37:58.845483 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.845457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a3599c3-0b33-4364-b635-5ab62887d0ac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4ff5n\" (UID: \"3a3599c3-0b33-4364-b635-5ab62887d0ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"
Sep 09 21:37:58.853252 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.853224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gg45\" (UniqueName: \"kubernetes.io/projected/3a3599c3-0b33-4364-b635-5ab62887d0ac-kube-api-access-4gg45\") pod \"openshift-lws-operator-fd8ccff4c-4ff5n\" (UID: \"3a3599c3-0b33-4364-b635-5ab62887d0ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"
Sep 09 21:37:58.952887 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:58.952793    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"
Sep 09 21:37:59.221578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:59.221417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n"]
Sep 09 21:37:59.224487 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:37:59.224460    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a3599c3_0b33_4364_b635_5ab62887d0ac.slice/crio-572faab61086ac60663fd297c3c6cb166b41f2ddfdd4f631709034c3b27f8c8c WatchSource:0}: Error finding container 572faab61086ac60663fd297c3c6cb166b41f2ddfdd4f631709034c3b27f8c8c: Status 404 returned error can't find the container with id 572faab61086ac60663fd297c3c6cb166b41f2ddfdd4f631709034c3b27f8c8c
Sep 09 21:37:59.359241 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:37:59.359201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n" event={"ID":"3a3599c3-0b33-4364-b635-5ab62887d0ac","Type":"ContainerStarted","Data":"572faab61086ac60663fd297c3c6cb166b41f2ddfdd4f631709034c3b27f8c8c"}
Sep 09 21:38:02.373881 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:02.373843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n" event={"ID":"3a3599c3-0b33-4364-b635-5ab62887d0ac","Type":"ContainerStarted","Data":"e2adb9b6119980eb6c09fd93a875c5cdd8ba006b162ab195d6618277d606eacd"}
Sep 09 21:38:02.390730 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:02.390674    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ff5n" podStartSLOduration=2.075742175 podStartE2EDuration="4.390658249s" totalImagesPullingTime="2.314916074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:02.389563778 +0000 UTC m=+287.875446812" watchObservedRunningTime="2026-09-09 21:38:02.390658249 +0000 UTC m=+287.876541287"
Sep 09 21:38:14.935599 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:14.935572    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 21:38:19.310825 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.310789    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"]
Sep 09 21:38:19.313888 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.313872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.317838 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.317816    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 21:38:19.317974 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.317876    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 21:38:19.317974 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.317922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 21:38:19.317974 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.317935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gxjm8\""
Sep 09 21:38:19.326259 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.326229    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"]
Sep 09 21:38:19.417202 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.417157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68cm\" (UniqueName: \"kubernetes.io/projected/247d11fe-12b8-414b-b3a4-59807401b9e0-kube-api-access-p68cm\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.417390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.417231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/247d11fe-12b8-414b-b3a4-59807401b9e0-metrics-cert\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.417390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.417258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/247d11fe-12b8-414b-b3a4-59807401b9e0-cert\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.417390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.417280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/247d11fe-12b8-414b-b3a4-59807401b9e0-manager-config\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.518157 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.518104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p68cm\" (UniqueName: \"kubernetes.io/projected/247d11fe-12b8-414b-b3a4-59807401b9e0-kube-api-access-p68cm\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.518361 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.518194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/247d11fe-12b8-414b-b3a4-59807401b9e0-metrics-cert\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.518361 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.518223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/247d11fe-12b8-414b-b3a4-59807401b9e0-cert\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.518361 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.518246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/247d11fe-12b8-414b-b3a4-59807401b9e0-manager-config\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.518919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.518898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/247d11fe-12b8-414b-b3a4-59807401b9e0-manager-config\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.520941 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.520921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/247d11fe-12b8-414b-b3a4-59807401b9e0-metrics-cert\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.521005 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.520943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/247d11fe-12b8-414b-b3a4-59807401b9e0-cert\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.526866 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.526836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68cm\" (UniqueName: \"kubernetes.io/projected/247d11fe-12b8-414b-b3a4-59807401b9e0-kube-api-access-p68cm\") pod \"lws-controller-manager-64d94cf4d-2t4b8\" (UID: \"247d11fe-12b8-414b-b3a4-59807401b9e0\") " pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.623478 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.623436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:19.899857 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.899795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"]
Sep 09 21:38:19.901943 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:38:19.901918    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247d11fe_12b8_414b_b3a4_59807401b9e0.slice/crio-1a83b8edeafa0dd3e9a84250ff42fb4d77c75e085b980af1c1071b40488be3ae WatchSource:0}: Error finding container 1a83b8edeafa0dd3e9a84250ff42fb4d77c75e085b980af1c1071b40488be3ae: Status 404 returned error can't find the container with id 1a83b8edeafa0dd3e9a84250ff42fb4d77c75e085b980af1c1071b40488be3ae
Sep 09 21:38:19.904002 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:19.903986    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:38:20.443155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:20.443118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8" event={"ID":"247d11fe-12b8-414b-b3a4-59807401b9e0","Type":"ContainerStarted","Data":"1a83b8edeafa0dd3e9a84250ff42fb4d77c75e085b980af1c1071b40488be3ae"}
Sep 09 21:38:21.706199 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.706167    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"]
Sep 09 21:38:21.709748 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.709708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.714013 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.713973    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9n6ch\""
Sep 09 21:38:21.714013 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.714001    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 21:38:21.714230 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.714032    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 21:38:21.714230 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.713974    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 21:38:21.714230 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.713968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 21:38:21.725321 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.725297    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"]
Sep 09 21:38:21.841198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.841111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-tmp\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.841198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.841148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.841198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.841178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-manifests\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.841421 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.841243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2qr\" (UniqueName: \"kubernetes.io/projected/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-kube-api-access-ld2qr\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.841421 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.841343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.942704 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.942659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-tmp\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.942704 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.942702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.942923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.942851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-manifests\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.942923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.942887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2qr\" (UniqueName: \"kubernetes.io/projected/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-kube-api-access-ld2qr\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.942988 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.942954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.943245 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.943213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-manifests\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.945243 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.945213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-tmp\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.945355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.945257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.945355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.945336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:21.951323 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:21.951298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2qr\" (UniqueName: \"kubernetes.io/projected/6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb-kube-api-access-ld2qr\") pod \"opendatahub-operator-controller-manager-84fbbd4d59-f6tcl\" (UID: \"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb\") " pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:22.022025 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:22.021987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:22.305896 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:22.305843    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"]
Sep 09 21:38:22.309890 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:38:22.309861    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf2871c_c446_4948_b9d1_8ac0b9d9d3bb.slice/crio-04b0017cfc15c0b651272a12487d3fc5c414d25a421447d97c7841f7461678f9 WatchSource:0}: Error finding container 04b0017cfc15c0b651272a12487d3fc5c414d25a421447d97c7841f7461678f9: Status 404 returned error can't find the container with id 04b0017cfc15c0b651272a12487d3fc5c414d25a421447d97c7841f7461678f9
Sep 09 21:38:22.454400 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:22.454364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8" event={"ID":"247d11fe-12b8-414b-b3a4-59807401b9e0","Type":"ContainerStarted","Data":"bda13e7b43caf0666287a4ddb2d09dc9d4562471cf7adc6a25f8a32c6bebb155"}
Sep 09 21:38:22.454593 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:22.454423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:22.455755 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:22.455703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl" event={"ID":"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb","Type":"ContainerStarted","Data":"04b0017cfc15c0b651272a12487d3fc5c414d25a421447d97c7841f7461678f9"}
Sep 09 21:38:22.472262 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:22.472197    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8" podStartSLOduration=1.843854414 podStartE2EDuration="3.472178473s" totalImagesPullingTime="1.628324059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:22.471472702 +0000 UTC m=+307.957355740" watchObservedRunningTime="2026-09-09 21:38:22.472178473 +0000 UTC m=+307.958061512"
Sep 09 21:38:26.474448 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:26.474404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl" event={"ID":"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb","Type":"ContainerStarted","Data":"d1944ff105bf6a039d11b1c1b96951e5d5552cb66259180dc3b2aaa7c1624db6"}
Sep 09 21:38:27.480788 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:27.480752    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb" containerID="d1944ff105bf6a039d11b1c1b96951e5d5552cb66259180dc3b2aaa7c1624db6" exitCode=0
Sep 09 21:38:27.481159 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:27.480811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl" event={"ID":"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb","Type":"ContainerDied","Data":"d1944ff105bf6a039d11b1c1b96951e5d5552cb66259180dc3b2aaa7c1624db6"}
Sep 09 21:38:28.487217 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:28.487129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl" event={"ID":"6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb","Type":"ContainerStarted","Data":"8d29f5008b61581810b6eb7b904837859df59f9dc1baa7630e2fee0e7ca160c6"}
Sep 09 21:38:28.487593 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:28.487282    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:28.512818 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:28.512763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl" podStartSLOduration=2.278762677 podStartE2EDuration="7.512745938s" totalImagesPullingTime="4.193755877s" totalInitContainerRuntime="1.040227384s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:28.511445553 +0000 UTC m=+313.997328596" watchObservedRunningTime="2026-09-09 21:38:28.512745938 +0000 UTC m=+313.998628976"
Sep 09 21:38:33.462899 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:33.462862    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-64d94cf4d-2t4b8"
Sep 09 21:38:39.493876 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:39.493836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84fbbd4d59-f6tcl"
Sep 09 21:38:41.401711 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.401672    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5769888876-6z8xd"]
Sep 09 21:38:41.406799 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.405545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.412658 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.412621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 21:38:41.412890 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.412843    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 21:38:41.412890 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.412872    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 21:38:41.413069 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.412635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pkqpx\""
Sep 09 21:38:41.413069 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.412989    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 21:38:41.425373 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.425346    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5769888876-6z8xd"]
Sep 09 21:38:41.515883 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.515848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c1023d-671f-4ae3-85c4-e52a4860a832-tmp\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.515883 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.515887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c1023d-671f-4ae3-85c4-e52a4860a832-tls-certs\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.516093 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.515965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ddl\" (UniqueName: \"kubernetes.io/projected/23c1023d-671f-4ae3-85c4-e52a4860a832-kube-api-access-b6ddl\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.616628 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.616589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ddl\" (UniqueName: \"kubernetes.io/projected/23c1023d-671f-4ae3-85c4-e52a4860a832-kube-api-access-b6ddl\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.616832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.616687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c1023d-671f-4ae3-85c4-e52a4860a832-tmp\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.616832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.616746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c1023d-671f-4ae3-85c4-e52a4860a832-tls-certs\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.619146 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.619120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c1023d-671f-4ae3-85c4-e52a4860a832-tmp\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.619367 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.619345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c1023d-671f-4ae3-85c4-e52a4860a832-tls-certs\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.625292 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.625264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ddl\" (UniqueName: \"kubernetes.io/projected/23c1023d-671f-4ae3-85c4-e52a4860a832-kube-api-access-b6ddl\") pod \"kube-auth-proxy-5769888876-6z8xd\" (UID: \"23c1023d-671f-4ae3-85c4-e52a4860a832\") " pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.716921 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.716837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd"
Sep 09 21:38:41.984224 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:41.984124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5769888876-6z8xd"]
Sep 09 21:38:41.986237 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:38:41.986215    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c1023d_671f_4ae3_85c4_e52a4860a832.slice/crio-c9d652c5b8a7c5711b4301725a40cf176364f8fa3d752efcc15ecb5ac77e5833 WatchSource:0}: Error finding container c9d652c5b8a7c5711b4301725a40cf176364f8fa3d752efcc15ecb5ac77e5833: Status 404 returned error can't find the container with id c9d652c5b8a7c5711b4301725a40cf176364f8fa3d752efcc15ecb5ac77e5833
Sep 09 21:38:42.551399 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:42.551354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd" event={"ID":"23c1023d-671f-4ae3-85c4-e52a4860a832","Type":"ContainerStarted","Data":"c9d652c5b8a7c5711b4301725a40cf176364f8fa3d752efcc15ecb5ac77e5833"}
Sep 09 21:38:45.571907 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:45.571866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd" event={"ID":"23c1023d-671f-4ae3-85c4-e52a4860a832","Type":"ContainerStarted","Data":"af3071d93c22d3edb5f0c47e41c5e6eff78d6015af63d982109c2c880edc4332"}
Sep 09 21:38:45.589476 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:38:45.589427    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5769888876-6z8xd" podStartSLOduration=1.172193341 podStartE2EDuration="4.589411487s" totalImagesPullingTime="3.417218146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:45.588942174 +0000 UTC m=+331.074825203" watchObservedRunningTime="2026-09-09 21:38:45.589411487 +0000 UTC m=+331.075294525"
Sep 09 21:39:39.644766 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.644674    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d6bff9dfc-jm2rw"]
Sep 09 21:39:39.648119 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.648093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.658501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.658469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d6bff9dfc-jm2rw"]
Sep 09 21:39:39.708077 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.708040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjz8m\" (UniqueName: \"kubernetes.io/projected/17af9ed6-06c2-4774-8e81-233c07648bb2-kube-api-access-bjz8m\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.708264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.708104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17af9ed6-06c2-4774-8e81-233c07648bb2-console-oauth-config\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.708264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.708141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-oauth-serving-cert\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.708264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.708169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-service-ca\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.708264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.708190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-trusted-ca-bundle\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.708264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.708230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-console-config\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.708264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.708261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17af9ed6-06c2-4774-8e81-233c07648bb2-console-serving-cert\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.808737 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.808692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjz8m\" (UniqueName: \"kubernetes.io/projected/17af9ed6-06c2-4774-8e81-233c07648bb2-kube-api-access-bjz8m\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.808943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.808797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17af9ed6-06c2-4774-8e81-233c07648bb2-console-oauth-config\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.808943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.808840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-oauth-serving-cert\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.808943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.808869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-service-ca\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.808943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.808890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-trusted-ca-bundle\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.809158 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.809017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-console-config\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.809158 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.809064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17af9ed6-06c2-4774-8e81-233c07648bb2-console-serving-cert\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.809808 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.809783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-service-ca\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.809808 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.809788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-oauth-serving-cert\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.810131 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.810103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-console-config\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.810252 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.810105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17af9ed6-06c2-4774-8e81-233c07648bb2-trusted-ca-bundle\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.811795 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.811770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17af9ed6-06c2-4774-8e81-233c07648bb2-console-oauth-config\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.811888 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.811850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17af9ed6-06c2-4774-8e81-233c07648bb2-console-serving-cert\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.819297 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.819277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjz8m\" (UniqueName: \"kubernetes.io/projected/17af9ed6-06c2-4774-8e81-233c07648bb2-kube-api-access-bjz8m\") pod \"console-5d6bff9dfc-jm2rw\" (UID: \"17af9ed6-06c2-4774-8e81-233c07648bb2\") " pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:39.958411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:39.958319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:40.232130 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:40.232105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d6bff9dfc-jm2rw"]
Sep 09 21:39:40.234184 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:39:40.234157    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17af9ed6_06c2_4774_8e81_233c07648bb2.slice/crio-808fedb3033a13188d6e3a461a8910fad4ef230420c9fe4c3beb8e6835639f1d WatchSource:0}: Error finding container 808fedb3033a13188d6e3a461a8910fad4ef230420c9fe4c3beb8e6835639f1d: Status 404 returned error can't find the container with id 808fedb3033a13188d6e3a461a8910fad4ef230420c9fe4c3beb8e6835639f1d
Sep 09 21:39:40.819153 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:40.819119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6bff9dfc-jm2rw" event={"ID":"17af9ed6-06c2-4774-8e81-233c07648bb2","Type":"ContainerStarted","Data":"39833e3e09751910d615213642be2dca66ee3dfe1dd4ab52fcde5d3c94fcff9d"}
Sep 09 21:39:40.819508 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:40.819165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6bff9dfc-jm2rw" event={"ID":"17af9ed6-06c2-4774-8e81-233c07648bb2","Type":"ContainerStarted","Data":"808fedb3033a13188d6e3a461a8910fad4ef230420c9fe4c3beb8e6835639f1d"}
Sep 09 21:39:40.838860 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:40.838807    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d6bff9dfc-jm2rw" podStartSLOduration=1.838792095 podStartE2EDuration="1.838792095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:40.838276346 +0000 UTC m=+386.324159383" watchObservedRunningTime="2026-09-09 21:39:40.838792095 +0000 UTC m=+386.324675133"
Sep 09 21:39:49.958748 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:49.958694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:49.959225 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:49.958771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:49.963763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:49.963712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:50.866358 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:50.866328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d6bff9dfc-jm2rw"
Sep 09 21:39:50.908613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:39:50.908583    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-97ffcbf56-gs4lf"]
Sep 09 21:40:15.929676 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:15.929625    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-97ffcbf56-gs4lf" podUID="292b56a0-6b1b-45b6-aa9d-6528ab91923b" containerName="console" containerID="cri-o://78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad" gracePeriod=15
Sep 09 21:40:15.954870 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:40:15.954834    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292b56a0_6b1b_45b6_aa9d_6528ab91923b.slice/crio-78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 21:40:16.168616 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.168593    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:40:16.340949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.340868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-service-ca\") pod \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") "
Sep 09 21:40:16.340949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.340907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-serving-cert\") pod \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") "
Sep 09 21:40:16.340949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.340945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlz5c\" (UniqueName: \"kubernetes.io/projected/292b56a0-6b1b-45b6-aa9d-6528ab91923b-kube-api-access-nlz5c\") pod \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") "
Sep 09 21:40:16.341238 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.341013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-oauth-serving-cert\") pod \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") "
Sep 09 21:40:16.341238 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.341051    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-trusted-ca-bundle\") pod \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") "
Sep 09 21:40:16.341238 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.341081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-config\") pod \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") "
Sep 09 21:40:16.341238 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.341125    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-oauth-config\") pod \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\" (UID: \"292b56a0-6b1b-45b6-aa9d-6528ab91923b\") "
Sep 09 21:40:16.341446 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.341415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-service-ca" (OuterVolumeSpecName: "service-ca") pod "292b56a0-6b1b-45b6-aa9d-6528ab91923b" (UID: "292b56a0-6b1b-45b6-aa9d-6528ab91923b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:40:16.341527 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.341496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "292b56a0-6b1b-45b6-aa9d-6528ab91923b" (UID: "292b56a0-6b1b-45b6-aa9d-6528ab91923b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:40:16.341527 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.341504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-config" (OuterVolumeSpecName: "console-config") pod "292b56a0-6b1b-45b6-aa9d-6528ab91923b" (UID: "292b56a0-6b1b-45b6-aa9d-6528ab91923b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:40:16.341621 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.341524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "292b56a0-6b1b-45b6-aa9d-6528ab91923b" (UID: "292b56a0-6b1b-45b6-aa9d-6528ab91923b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:40:16.343298 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.343268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "292b56a0-6b1b-45b6-aa9d-6528ab91923b" (UID: "292b56a0-6b1b-45b6-aa9d-6528ab91923b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:40:16.343298 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.343290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292b56a0-6b1b-45b6-aa9d-6528ab91923b-kube-api-access-nlz5c" (OuterVolumeSpecName: "kube-api-access-nlz5c") pod "292b56a0-6b1b-45b6-aa9d-6528ab91923b" (UID: "292b56a0-6b1b-45b6-aa9d-6528ab91923b"). InnerVolumeSpecName "kube-api-access-nlz5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:40:16.343433 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.343336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "292b56a0-6b1b-45b6-aa9d-6528ab91923b" (UID: "292b56a0-6b1b-45b6-aa9d-6528ab91923b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:40:16.442260 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.442218    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.442260 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.442252    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlz5c\" (UniqueName: \"kubernetes.io/projected/292b56a0-6b1b-45b6-aa9d-6528ab91923b-kube-api-access-nlz5c\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.442260 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.442262    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-oauth-serving-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.442481 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.442271    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-trusted-ca-bundle\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.442481 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.442281    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.442481 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.442289    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/292b56a0-6b1b-45b6-aa9d-6528ab91923b-console-oauth-config\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.442481 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.442297    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/292b56a0-6b1b-45b6-aa9d-6528ab91923b-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.985658 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.985622    2563 generic.go:356] "Generic (PLEG): container finished" podID="292b56a0-6b1b-45b6-aa9d-6528ab91923b" containerID="78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad" exitCode=0
Sep 09 21:40:16.986053 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.985670    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97ffcbf56-gs4lf"
Sep 09 21:40:16.986053 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.985707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97ffcbf56-gs4lf" event={"ID":"292b56a0-6b1b-45b6-aa9d-6528ab91923b","Type":"ContainerDied","Data":"78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad"}
Sep 09 21:40:16.986053 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.985772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97ffcbf56-gs4lf" event={"ID":"292b56a0-6b1b-45b6-aa9d-6528ab91923b","Type":"ContainerDied","Data":"503c9fa12db0c6dc69b8a8bc67e174d636ebc059924e1426368883af8582c95e"}
Sep 09 21:40:16.986053 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.985792    2563 scope.go:117] "RemoveContainer" containerID="78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad"
Sep 09 21:40:16.994375 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.994358    2563 scope.go:117] "RemoveContainer" containerID="78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad"
Sep 09 21:40:16.994644 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:40:16.994628    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad\": container with ID starting with 78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad not found: ID does not exist" containerID="78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad"
Sep 09 21:40:16.994681 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:16.994653    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad"} err="failed to get container status \"78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad\": rpc error: code = NotFound desc = could not find container \"78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad\": container with ID starting with 78bbc917ab8a5548ed34111605530a4c21e2dbade6ef766cb4dc345962ee30ad not found: ID does not exist"
Sep 09 21:40:17.007154 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:17.007129    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-97ffcbf56-gs4lf"]
Sep 09 21:40:17.013095 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:17.013070    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-97ffcbf56-gs4lf"]
Sep 09 21:40:17.046097 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:40:17.046070    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="292b56a0-6b1b-45b6-aa9d-6528ab91923b" path="/var/lib/kubelet/pods/292b56a0-6b1b-45b6-aa9d-6528ab91923b/volumes"
Sep 09 21:41:01.151154 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.151119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6fbmp"]
Sep 09 21:41:01.151605 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.151488    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292b56a0-6b1b-45b6-aa9d-6528ab91923b" containerName="console"
Sep 09 21:41:01.151605 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.151500    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="292b56a0-6b1b-45b6-aa9d-6528ab91923b" containerName="console"
Sep 09 21:41:01.151605 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.151564    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="292b56a0-6b1b-45b6-aa9d-6528ab91923b" containerName="console"
Sep 09 21:41:01.154619 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.154601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:01.157241 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.157213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 21:41:01.157367 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.157245    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 21:41:01.157367 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.157212    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 21:41:01.158362 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.158342    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4r4xn\""
Sep 09 21:41:01.161938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.161760    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6fbmp"]
Sep 09 21:41:01.237812 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.237769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45hv\" (UniqueName: \"kubernetes.io/projected/56ad5b76-0d13-4fb7-961d-ff0e37960656-kube-api-access-j45hv\") pod \"limitador-limitador-8588749977-6fbmp\" (UID: \"56ad5b76-0d13-4fb7-961d-ff0e37960656\") " pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:01.237968 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.237857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ad5b76-0d13-4fb7-961d-ff0e37960656-config-file\") pod \"limitador-limitador-8588749977-6fbmp\" (UID: \"56ad5b76-0d13-4fb7-961d-ff0e37960656\") " pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:01.247253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.247222    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6fbmp"]
Sep 09 21:41:01.338826 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.338795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45hv\" (UniqueName: \"kubernetes.io/projected/56ad5b76-0d13-4fb7-961d-ff0e37960656-kube-api-access-j45hv\") pod \"limitador-limitador-8588749977-6fbmp\" (UID: \"56ad5b76-0d13-4fb7-961d-ff0e37960656\") " pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:01.339008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.338849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ad5b76-0d13-4fb7-961d-ff0e37960656-config-file\") pod \"limitador-limitador-8588749977-6fbmp\" (UID: \"56ad5b76-0d13-4fb7-961d-ff0e37960656\") " pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:01.339413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.339396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ad5b76-0d13-4fb7-961d-ff0e37960656-config-file\") pod \"limitador-limitador-8588749977-6fbmp\" (UID: \"56ad5b76-0d13-4fb7-961d-ff0e37960656\") " pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:01.346786 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.346756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45hv\" (UniqueName: \"kubernetes.io/projected/56ad5b76-0d13-4fb7-961d-ff0e37960656-kube-api-access-j45hv\") pod \"limitador-limitador-8588749977-6fbmp\" (UID: \"56ad5b76-0d13-4fb7-961d-ff0e37960656\") " pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:01.466077 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.465979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:01.734380 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.734354    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6fbmp"]
Sep 09 21:41:01.736767 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:41:01.736741    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ad5b76_0d13_4fb7_961d_ff0e37960656.slice/crio-e848f0c46d1cc4b3dd722f1ab97904dc6ce212081c604e50ea534aed76e2d3c0 WatchSource:0}: Error finding container e848f0c46d1cc4b3dd722f1ab97904dc6ce212081c604e50ea534aed76e2d3c0: Status 404 returned error can't find the container with id e848f0c46d1cc4b3dd722f1ab97904dc6ce212081c604e50ea534aed76e2d3c0
Sep 09 21:41:01.946462 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.946420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nb8fw"]
Sep 09 21:41:01.951315 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.951287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw"
Sep 09 21:41:01.953910 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.953886    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9qdlq\""
Sep 09 21:41:01.956668 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:01.956645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nb8fw"]
Sep 09 21:41:02.045683 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.045584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnplt\" (UniqueName: \"kubernetes.io/projected/036103eb-3a89-4a8c-8689-5040000c40b4-kube-api-access-gnplt\") pod \"authorino-7bcfd8f9cd-nb8fw\" (UID: \"036103eb-3a89-4a8c-8689-5040000c40b4\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw"
Sep 09 21:41:02.060360 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.060326    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-fdpts"]
Sep 09 21:41:02.063653 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.063636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fdpts"
Sep 09 21:41:02.069542 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.069517    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fdpts"]
Sep 09 21:41:02.146048 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.146018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnplt\" (UniqueName: \"kubernetes.io/projected/036103eb-3a89-4a8c-8689-5040000c40b4-kube-api-access-gnplt\") pod \"authorino-7bcfd8f9cd-nb8fw\" (UID: \"036103eb-3a89-4a8c-8689-5040000c40b4\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw"
Sep 09 21:41:02.154583 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.154554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnplt\" (UniqueName: \"kubernetes.io/projected/036103eb-3a89-4a8c-8689-5040000c40b4-kube-api-access-gnplt\") pod \"authorino-7bcfd8f9cd-nb8fw\" (UID: \"036103eb-3a89-4a8c-8689-5040000c40b4\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw"
Sep 09 21:41:02.194079 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.194049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp" event={"ID":"56ad5b76-0d13-4fb7-961d-ff0e37960656","Type":"ContainerStarted","Data":"e848f0c46d1cc4b3dd722f1ab97904dc6ce212081c604e50ea534aed76e2d3c0"}
Sep 09 21:41:02.247451 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.247418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxwp\" (UniqueName: \"kubernetes.io/projected/a8db912c-a862-4940-b690-b85fc3f09c6c-kube-api-access-qfxwp\") pod \"authorino-59c974bd48-fdpts\" (UID: \"a8db912c-a862-4940-b690-b85fc3f09c6c\") " pod="kuadrant-system/authorino-59c974bd48-fdpts"
Sep 09 21:41:02.262337 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.262312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw"
Sep 09 21:41:02.348079 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.348044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxwp\" (UniqueName: \"kubernetes.io/projected/a8db912c-a862-4940-b690-b85fc3f09c6c-kube-api-access-qfxwp\") pod \"authorino-59c974bd48-fdpts\" (UID: \"a8db912c-a862-4940-b690-b85fc3f09c6c\") " pod="kuadrant-system/authorino-59c974bd48-fdpts"
Sep 09 21:41:02.357359 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.357333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxwp\" (UniqueName: \"kubernetes.io/projected/a8db912c-a862-4940-b690-b85fc3f09c6c-kube-api-access-qfxwp\") pod \"authorino-59c974bd48-fdpts\" (UID: \"a8db912c-a862-4940-b690-b85fc3f09c6c\") " pod="kuadrant-system/authorino-59c974bd48-fdpts"
Sep 09 21:41:02.374301 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.374265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fdpts"
Sep 09 21:41:02.528968 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.528942    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nb8fw"]
Sep 09 21:41:02.531074 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:41:02.531047    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod036103eb_3a89_4a8c_8689_5040000c40b4.slice/crio-a1046c9e98445c83efa91b3b392cb66fa043eaebaa3d83166ea8902b3d7a7d99 WatchSource:0}: Error finding container a1046c9e98445c83efa91b3b392cb66fa043eaebaa3d83166ea8902b3d7a7d99: Status 404 returned error can't find the container with id a1046c9e98445c83efa91b3b392cb66fa043eaebaa3d83166ea8902b3d7a7d99
Sep 09 21:41:02.645033 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:02.645008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fdpts"]
Sep 09 21:41:02.646987 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:41:02.646961    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8db912c_a862_4940_b690_b85fc3f09c6c.slice/crio-37f829785f92b768eca63cc8fc0d9f09712c53a02b6b9e243ce58f1f0f648c6c WatchSource:0}: Error finding container 37f829785f92b768eca63cc8fc0d9f09712c53a02b6b9e243ce58f1f0f648c6c: Status 404 returned error can't find the container with id 37f829785f92b768eca63cc8fc0d9f09712c53a02b6b9e243ce58f1f0f648c6c
Sep 09 21:41:03.201618 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:03.201585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fdpts" event={"ID":"a8db912c-a862-4940-b690-b85fc3f09c6c","Type":"ContainerStarted","Data":"37f829785f92b768eca63cc8fc0d9f09712c53a02b6b9e243ce58f1f0f648c6c"}
Sep 09 21:41:03.203070 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:03.203042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw" event={"ID":"036103eb-3a89-4a8c-8689-5040000c40b4","Type":"ContainerStarted","Data":"a1046c9e98445c83efa91b3b392cb66fa043eaebaa3d83166ea8902b3d7a7d99"}
Sep 09 21:41:06.221615 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:06.221577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fdpts" event={"ID":"a8db912c-a862-4940-b690-b85fc3f09c6c","Type":"ContainerStarted","Data":"43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4"}
Sep 09 21:41:06.223050 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:06.223023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw" event={"ID":"036103eb-3a89-4a8c-8689-5040000c40b4","Type":"ContainerStarted","Data":"bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65"}
Sep 09 21:41:06.224417 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:06.224394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp" event={"ID":"56ad5b76-0d13-4fb7-961d-ff0e37960656","Type":"ContainerStarted","Data":"0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6"}
Sep 09 21:41:06.224502 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:06.224473    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:06.236178 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:06.236134    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-fdpts" podStartSLOduration=1.252373487 podStartE2EDuration="4.236119184s" totalImagesPullingTime="2.983745697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:06.235005517 +0000 UTC m=+471.720888554" watchObservedRunningTime="2026-09-09 21:41:06.236119184 +0000 UTC m=+471.722002222"
Sep 09 21:41:06.250310 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:06.250254    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp" podStartSLOduration=1.3698922 podStartE2EDuration="5.250239246s" totalImagesPullingTime="3.880347046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:06.249914813 +0000 UTC m=+471.735797875" watchObservedRunningTime="2026-09-09 21:41:06.250239246 +0000 UTC m=+471.736122285"
Sep 09 21:41:06.264133 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:06.264105    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nb8fw"]
Sep 09 21:41:06.274454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:06.274409    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw" podStartSLOduration=2.186861767 podStartE2EDuration="5.274391881s" totalImagesPullingTime="3.087530114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:06.272849856 +0000 UTC m=+471.758732895" watchObservedRunningTime="2026-09-09 21:41:06.274391881 +0000 UTC m=+471.760274919"
Sep 09 21:41:08.234298 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:08.234255    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw" podUID="036103eb-3a89-4a8c-8689-5040000c40b4" containerName="authorino" containerID="cri-o://bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65" gracePeriod=30
Sep 09 21:41:08.478697 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:08.478671    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw"
Sep 09 21:41:08.506885 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:08.506795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnplt\" (UniqueName: \"kubernetes.io/projected/036103eb-3a89-4a8c-8689-5040000c40b4-kube-api-access-gnplt\") pod \"036103eb-3a89-4a8c-8689-5040000c40b4\" (UID: \"036103eb-3a89-4a8c-8689-5040000c40b4\") "
Sep 09 21:41:08.509105 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:08.509077    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036103eb-3a89-4a8c-8689-5040000c40b4-kube-api-access-gnplt" (OuterVolumeSpecName: "kube-api-access-gnplt") pod "036103eb-3a89-4a8c-8689-5040000c40b4" (UID: "036103eb-3a89-4a8c-8689-5040000c40b4"). InnerVolumeSpecName "kube-api-access-gnplt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:08.608175 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:08.608137    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnplt\" (UniqueName: \"kubernetes.io/projected/036103eb-3a89-4a8c-8689-5040000c40b4-kube-api-access-gnplt\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:41:09.239675 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.239638    2563 generic.go:356] "Generic (PLEG): container finished" podID="036103eb-3a89-4a8c-8689-5040000c40b4" containerID="bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65" exitCode=0
Sep 09 21:41:09.240176 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.239686    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw"
Sep 09 21:41:09.240176 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.239748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw" event={"ID":"036103eb-3a89-4a8c-8689-5040000c40b4","Type":"ContainerDied","Data":"bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65"}
Sep 09 21:41:09.240176 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.239792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nb8fw" event={"ID":"036103eb-3a89-4a8c-8689-5040000c40b4","Type":"ContainerDied","Data":"a1046c9e98445c83efa91b3b392cb66fa043eaebaa3d83166ea8902b3d7a7d99"}
Sep 09 21:41:09.240176 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.239811    2563 scope.go:117] "RemoveContainer" containerID="bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65"
Sep 09 21:41:09.248237 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.248217    2563 scope.go:117] "RemoveContainer" containerID="bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65"
Sep 09 21:41:09.248490 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:41:09.248472    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65\": container with ID starting with bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65 not found: ID does not exist" containerID="bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65"
Sep 09 21:41:09.248547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.248496    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65"} err="failed to get container status \"bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65\": rpc error: code = NotFound desc = could not find container \"bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65\": container with ID starting with bec05dc58f99a2fe3bcad7e127c4eccc8a06094bf88678c644f7f49d9937eb65 not found: ID does not exist"
Sep 09 21:41:09.254877 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.254856    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nb8fw"]
Sep 09 21:41:09.258292 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:09.258270    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nb8fw"]
Sep 09 21:41:11.045964 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:11.045931    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="036103eb-3a89-4a8c-8689-5040000c40b4" path="/var/lib/kubelet/pods/036103eb-3a89-4a8c-8689-5040000c40b4/volumes"
Sep 09 21:41:15.262494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:15.262409    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6fbmp"]
Sep 09 21:41:15.262917 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:15.262589    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp" podUID="56ad5b76-0d13-4fb7-961d-ff0e37960656" containerName="limitador" containerID="cri-o://0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6" gracePeriod=30
Sep 09 21:41:15.263246 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:15.263225    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:16.234786 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.234761    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:16.269020 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.268993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ad5b76-0d13-4fb7-961d-ff0e37960656-config-file\") pod \"56ad5b76-0d13-4fb7-961d-ff0e37960656\" (UID: \"56ad5b76-0d13-4fb7-961d-ff0e37960656\") "
Sep 09 21:41:16.269451 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.269089    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j45hv\" (UniqueName: \"kubernetes.io/projected/56ad5b76-0d13-4fb7-961d-ff0e37960656-kube-api-access-j45hv\") pod \"56ad5b76-0d13-4fb7-961d-ff0e37960656\" (UID: \"56ad5b76-0d13-4fb7-961d-ff0e37960656\") "
Sep 09 21:41:16.269451 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.269355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ad5b76-0d13-4fb7-961d-ff0e37960656-config-file" (OuterVolumeSpecName: "config-file") pod "56ad5b76-0d13-4fb7-961d-ff0e37960656" (UID: "56ad5b76-0d13-4fb7-961d-ff0e37960656"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:41:16.271862 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.271833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ad5b76-0d13-4fb7-961d-ff0e37960656-kube-api-access-j45hv" (OuterVolumeSpecName: "kube-api-access-j45hv") pod "56ad5b76-0d13-4fb7-961d-ff0e37960656" (UID: "56ad5b76-0d13-4fb7-961d-ff0e37960656"). InnerVolumeSpecName "kube-api-access-j45hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:16.273323 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.273294    2563 generic.go:356] "Generic (PLEG): container finished" podID="56ad5b76-0d13-4fb7-961d-ff0e37960656" containerID="0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6" exitCode=0
Sep 09 21:41:16.273436 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.273342    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp"
Sep 09 21:41:16.273436 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.273370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp" event={"ID":"56ad5b76-0d13-4fb7-961d-ff0e37960656","Type":"ContainerDied","Data":"0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6"}
Sep 09 21:41:16.273436 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.273426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6fbmp" event={"ID":"56ad5b76-0d13-4fb7-961d-ff0e37960656","Type":"ContainerDied","Data":"e848f0c46d1cc4b3dd722f1ab97904dc6ce212081c604e50ea534aed76e2d3c0"}
Sep 09 21:41:16.273558 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.273450    2563 scope.go:117] "RemoveContainer" containerID="0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6"
Sep 09 21:41:16.285350 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.285329    2563 scope.go:117] "RemoveContainer" containerID="0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6"
Sep 09 21:41:16.285735 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:41:16.285689    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6\": container with ID starting with 0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6 not found: ID does not exist" containerID="0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6"
Sep 09 21:41:16.285810 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.285775    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6"} err="failed to get container status \"0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6\": rpc error: code = NotFound desc = could not find container \"0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6\": container with ID starting with 0720be7c8c145a885df450032a7f099b53981237e22318786d0f3974cc462cf6 not found: ID does not exist"
Sep 09 21:41:16.301880 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.301845    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6fbmp"]
Sep 09 21:41:16.302059 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.302024    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6fbmp"]
Sep 09 21:41:16.370703 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.370664    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j45hv\" (UniqueName: \"kubernetes.io/projected/56ad5b76-0d13-4fb7-961d-ff0e37960656-kube-api-access-j45hv\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:41:16.370703 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:16.370701    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56ad5b76-0d13-4fb7-961d-ff0e37960656-config-file\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:41:17.046149 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.046115    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ad5b76-0d13-4fb7-961d-ff0e37960656" path="/var/lib/kubelet/pods/56ad5b76-0d13-4fb7-961d-ff0e37960656/volumes"
Sep 09 21:41:17.638188 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.638150    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"]
Sep 09 21:41:17.638554 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.638520    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ad5b76-0d13-4fb7-961d-ff0e37960656" containerName="limitador"
Sep 09 21:41:17.638554 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.638533    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ad5b76-0d13-4fb7-961d-ff0e37960656" containerName="limitador"
Sep 09 21:41:17.638554 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.638543    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="036103eb-3a89-4a8c-8689-5040000c40b4" containerName="authorino"
Sep 09 21:41:17.638554 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.638549    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="036103eb-3a89-4a8c-8689-5040000c40b4" containerName="authorino"
Sep 09 21:41:17.638674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.638612    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="036103eb-3a89-4a8c-8689-5040000c40b4" containerName="authorino"
Sep 09 21:41:17.638674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.638622    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ad5b76-0d13-4fb7-961d-ff0e37960656" containerName="limitador"
Sep 09 21:41:17.643091 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.643070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:17.647113 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.647083    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-c6zvt\""
Sep 09 21:41:17.647235 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.647188    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:41:17.648255 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.648233    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 21:41:17.648382 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.648364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:41:17.657343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.657321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"]
Sep 09 21:41:17.682030 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.682005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2bba4bd-26c6-4858-8e05-c65abf81c4d3-data\") pod \"postgres-69fb4594cf-d6dp4\" (UID: \"a2bba4bd-26c6-4858-8e05-c65abf81c4d3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:17.682162 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.682051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4tms\" (UniqueName: \"kubernetes.io/projected/a2bba4bd-26c6-4858-8e05-c65abf81c4d3-kube-api-access-r4tms\") pod \"postgres-69fb4594cf-d6dp4\" (UID: \"a2bba4bd-26c6-4858-8e05-c65abf81c4d3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:17.782979 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.782944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2bba4bd-26c6-4858-8e05-c65abf81c4d3-data\") pod \"postgres-69fb4594cf-d6dp4\" (UID: \"a2bba4bd-26c6-4858-8e05-c65abf81c4d3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:17.782979 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.782983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4tms\" (UniqueName: \"kubernetes.io/projected/a2bba4bd-26c6-4858-8e05-c65abf81c4d3-kube-api-access-r4tms\") pod \"postgres-69fb4594cf-d6dp4\" (UID: \"a2bba4bd-26c6-4858-8e05-c65abf81c4d3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:17.783323 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.783306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2bba4bd-26c6-4858-8e05-c65abf81c4d3-data\") pod \"postgres-69fb4594cf-d6dp4\" (UID: \"a2bba4bd-26c6-4858-8e05-c65abf81c4d3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:17.794549 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.794522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4tms\" (UniqueName: \"kubernetes.io/projected/a2bba4bd-26c6-4858-8e05-c65abf81c4d3-kube-api-access-r4tms\") pod \"postgres-69fb4594cf-d6dp4\" (UID: \"a2bba4bd-26c6-4858-8e05-c65abf81c4d3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:17.953202 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:17.953116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:18.225537 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:18.225456    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"]
Sep 09 21:41:18.231537 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:41:18.231503    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bba4bd_26c6_4858_8e05_c65abf81c4d3.slice/crio-b46b7898700e4a2f775b7c8a4239a32a5717cd9a2b7937d7ef95b62897e5df04 WatchSource:0}: Error finding container b46b7898700e4a2f775b7c8a4239a32a5717cd9a2b7937d7ef95b62897e5df04: Status 404 returned error can't find the container with id b46b7898700e4a2f775b7c8a4239a32a5717cd9a2b7937d7ef95b62897e5df04
Sep 09 21:41:18.285086 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:18.285047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4" event={"ID":"a2bba4bd-26c6-4858-8e05-c65abf81c4d3","Type":"ContainerStarted","Data":"b46b7898700e4a2f775b7c8a4239a32a5717cd9a2b7937d7ef95b62897e5df04"}
Sep 09 21:41:23.313552 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:23.313462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4" event={"ID":"a2bba4bd-26c6-4858-8e05-c65abf81c4d3","Type":"ContainerStarted","Data":"a176e4f674ebdf801fd2cda5d0ba8edce9cc503a7e8ee6d0273f8374237f4542"}
Sep 09 21:41:23.313958 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:23.313591    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:23.329121 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:23.329062    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4" podStartSLOduration=1.607715125 podStartE2EDuration="6.329042961s" totalImagesPullingTime="4.721327836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:23.328920174 +0000 UTC m=+488.814803214" watchObservedRunningTime="2026-09-09 21:41:23.329042961 +0000 UTC m=+488.814926000"
Sep 09 21:41:29.348312 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.348277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d6dp4"
Sep 09 21:41:29.862413 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.862376    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-qvklb"]
Sep 09 21:41:29.868482 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.868456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qvklb"
Sep 09 21:41:29.872096 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.872057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-qvklb"]
Sep 09 21:41:29.919360 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.919328    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qvklb"]
Sep 09 21:41:29.919594 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:41:29.919575    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-46xtc], unattached volumes=[], failed to process volumes=[kube-api-access-46xtc]: context canceled" pod="kuadrant-system/authorino-557f876f69-qvklb" podUID="53d510ac-f4c9-4d26-aadc-7565bdb95873"
Sep 09 21:41:29.944911 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.944875    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p2dz6"]
Sep 09 21:41:29.948562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.948545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:29.951106 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.951084    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 21:41:29.954647 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.954627    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p2dz6"]
Sep 09 21:41:29.997351 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.997316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-tls-cert\") pod \"authorino-6d4dfc649f-p2dz6\" (UID: \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\") " pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:29.997351 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.997353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xtc\" (UniqueName: \"kubernetes.io/projected/53d510ac-f4c9-4d26-aadc-7565bdb95873-kube-api-access-46xtc\") pod \"authorino-557f876f69-qvklb\" (UID: \"53d510ac-f4c9-4d26-aadc-7565bdb95873\") " pod="kuadrant-system/authorino-557f876f69-qvklb"
Sep 09 21:41:29.997566 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:29.997383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrh4\" (UniqueName: \"kubernetes.io/projected/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-kube-api-access-cdrh4\") pod \"authorino-6d4dfc649f-p2dz6\" (UID: \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\") " pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:30.087996 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.087961    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p2dz6"]
Sep 09 21:41:30.088242 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:41:30.088222    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cdrh4 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-p2dz6" podUID="c5ea0ef9-f254-4efd-be06-b2d4927f1b72"
Sep 09 21:41:30.097871 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.097845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-tls-cert\") pod \"authorino-6d4dfc649f-p2dz6\" (UID: \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\") " pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:30.097999 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.097875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xtc\" (UniqueName: \"kubernetes.io/projected/53d510ac-f4c9-4d26-aadc-7565bdb95873-kube-api-access-46xtc\") pod \"authorino-557f876f69-qvklb\" (UID: \"53d510ac-f4c9-4d26-aadc-7565bdb95873\") " pod="kuadrant-system/authorino-557f876f69-qvklb"
Sep 09 21:41:30.097999 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.097910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrh4\" (UniqueName: \"kubernetes.io/projected/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-kube-api-access-cdrh4\") pod \"authorino-6d4dfc649f-p2dz6\" (UID: \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\") " pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:30.100567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.100542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-tls-cert\") pod \"authorino-6d4dfc649f-p2dz6\" (UID: \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\") " pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:30.111360 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.111337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrh4\" (UniqueName: \"kubernetes.io/projected/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-kube-api-access-cdrh4\") pod \"authorino-6d4dfc649f-p2dz6\" (UID: \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\") " pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:30.111614 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.111592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xtc\" (UniqueName: \"kubernetes.io/projected/53d510ac-f4c9-4d26-aadc-7565bdb95873-kube-api-access-46xtc\") pod \"authorino-557f876f69-qvklb\" (UID: \"53d510ac-f4c9-4d26-aadc-7565bdb95873\") " pod="kuadrant-system/authorino-557f876f69-qvklb"
Sep 09 21:41:30.115451 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.115394    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54bb746d97-8ghgr"]
Sep 09 21:41:30.119185 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.119164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:41:30.125366 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.125344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54bb746d97-8ghgr"]
Sep 09 21:41:30.198872 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.198832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d2d553d-0073-4423-a2e0-017791334ac4-tls-cert\") pod \"authorino-54bb746d97-8ghgr\" (UID: \"3d2d553d-0073-4423-a2e0-017791334ac4\") " pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:41:30.199044 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.198888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pz2\" (UniqueName: \"kubernetes.io/projected/3d2d553d-0073-4423-a2e0-017791334ac4-kube-api-access-p7pz2\") pod \"authorino-54bb746d97-8ghgr\" (UID: \"3d2d553d-0073-4423-a2e0-017791334ac4\") " pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:41:30.300317 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.300279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d2d553d-0073-4423-a2e0-017791334ac4-tls-cert\") pod \"authorino-54bb746d97-8ghgr\" (UID: \"3d2d553d-0073-4423-a2e0-017791334ac4\") " pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:41:30.300518 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.300341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pz2\" (UniqueName: \"kubernetes.io/projected/3d2d553d-0073-4423-a2e0-017791334ac4-kube-api-access-p7pz2\") pod \"authorino-54bb746d97-8ghgr\" (UID: \"3d2d553d-0073-4423-a2e0-017791334ac4\") " pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:41:30.302938 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.302914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d2d553d-0073-4423-a2e0-017791334ac4-tls-cert\") pod \"authorino-54bb746d97-8ghgr\" (UID: \"3d2d553d-0073-4423-a2e0-017791334ac4\") " pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:41:30.309232 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.309205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pz2\" (UniqueName: \"kubernetes.io/projected/3d2d553d-0073-4423-a2e0-017791334ac4-kube-api-access-p7pz2\") pod \"authorino-54bb746d97-8ghgr\" (UID: \"3d2d553d-0073-4423-a2e0-017791334ac4\") " pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:41:30.345908 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.345870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qvklb"
Sep 09 21:41:30.346098 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.345870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:30.353675 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.353649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qvklb"
Sep 09 21:41:30.356405 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.356389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:30.401229 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.401135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdrh4\" (UniqueName: \"kubernetes.io/projected/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-kube-api-access-cdrh4\") pod \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\" (UID: \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\") "
Sep 09 21:41:30.401229 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.401185    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46xtc\" (UniqueName: \"kubernetes.io/projected/53d510ac-f4c9-4d26-aadc-7565bdb95873-kube-api-access-46xtc\") pod \"53d510ac-f4c9-4d26-aadc-7565bdb95873\" (UID: \"53d510ac-f4c9-4d26-aadc-7565bdb95873\") "
Sep 09 21:41:30.401420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.401304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-tls-cert\") pod \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\" (UID: \"c5ea0ef9-f254-4efd-be06-b2d4927f1b72\") "
Sep 09 21:41:30.403541 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.403510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-kube-api-access-cdrh4" (OuterVolumeSpecName: "kube-api-access-cdrh4") pod "c5ea0ef9-f254-4efd-be06-b2d4927f1b72" (UID: "c5ea0ef9-f254-4efd-be06-b2d4927f1b72"). InnerVolumeSpecName "kube-api-access-cdrh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:30.403674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.403540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c5ea0ef9-f254-4efd-be06-b2d4927f1b72" (UID: "c5ea0ef9-f254-4efd-be06-b2d4927f1b72"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:30.403674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.403540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d510ac-f4c9-4d26-aadc-7565bdb95873-kube-api-access-46xtc" (OuterVolumeSpecName: "kube-api-access-46xtc") pod "53d510ac-f4c9-4d26-aadc-7565bdb95873" (UID: "53d510ac-f4c9-4d26-aadc-7565bdb95873"). InnerVolumeSpecName "kube-api-access-46xtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:30.432857 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.432828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:41:30.502710 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.502671    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-tls-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:41:30.502710 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.502704    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrh4\" (UniqueName: \"kubernetes.io/projected/c5ea0ef9-f254-4efd-be06-b2d4927f1b72-kube-api-access-cdrh4\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:41:30.502909 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.502733    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46xtc\" (UniqueName: \"kubernetes.io/projected/53d510ac-f4c9-4d26-aadc-7565bdb95873-kube-api-access-46xtc\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:41:30.703304 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:30.703154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54bb746d97-8ghgr"]
Sep 09 21:41:30.705507 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:41:30.705480    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d2d553d_0073_4423_a2e0_017791334ac4.slice/crio-658f3cbc69e0050686e7b1b107669c9ef9e9d176e126ffbe875be1c86510c344 WatchSource:0}: Error finding container 658f3cbc69e0050686e7b1b107669c9ef9e9d176e126ffbe875be1c86510c344: Status 404 returned error can't find the container with id 658f3cbc69e0050686e7b1b107669c9ef9e9d176e126ffbe875be1c86510c344
Sep 09 21:41:31.351998 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.351962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54bb746d97-8ghgr" event={"ID":"3d2d553d-0073-4423-a2e0-017791334ac4","Type":"ContainerStarted","Data":"c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb"}
Sep 09 21:41:31.351998 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.351998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p2dz6"
Sep 09 21:41:31.352219 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.352009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54bb746d97-8ghgr" event={"ID":"3d2d553d-0073-4423-a2e0-017791334ac4","Type":"ContainerStarted","Data":"658f3cbc69e0050686e7b1b107669c9ef9e9d176e126ffbe875be1c86510c344"}
Sep 09 21:41:31.352219 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.351998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qvklb"
Sep 09 21:41:31.370184 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.370137    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54bb746d97-8ghgr" podStartSLOduration=1.035311975 podStartE2EDuration="1.370121035s" totalImagesPullingTime="334.80906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:31.369271088 +0000 UTC m=+496.855154127" watchObservedRunningTime="2026-09-09 21:41:31.370121035 +0000 UTC m=+496.856004087"
Sep 09 21:41:31.392530 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.392500    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qvklb"]
Sep 09 21:41:31.395124 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.395095    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fdpts"]
Sep 09 21:41:31.395336 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.395310    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-fdpts" podUID="a8db912c-a862-4940-b690-b85fc3f09c6c" containerName="authorino" containerID="cri-o://43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4" gracePeriod=30
Sep 09 21:41:31.396623 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.396601    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-qvklb"]
Sep 09 21:41:31.422952 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.422831    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p2dz6"]
Sep 09 21:41:31.430139 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.426071    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p2dz6"]
Sep 09 21:41:31.641451 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.641430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fdpts"
Sep 09 21:41:31.713291 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.713261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfxwp\" (UniqueName: \"kubernetes.io/projected/a8db912c-a862-4940-b690-b85fc3f09c6c-kube-api-access-qfxwp\") pod \"a8db912c-a862-4940-b690-b85fc3f09c6c\" (UID: \"a8db912c-a862-4940-b690-b85fc3f09c6c\") "
Sep 09 21:41:31.715506 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.715474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8db912c-a862-4940-b690-b85fc3f09c6c-kube-api-access-qfxwp" (OuterVolumeSpecName: "kube-api-access-qfxwp") pod "a8db912c-a862-4940-b690-b85fc3f09c6c" (UID: "a8db912c-a862-4940-b690-b85fc3f09c6c"). InnerVolumeSpecName "kube-api-access-qfxwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:31.814730 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:31.814681    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxwp\" (UniqueName: \"kubernetes.io/projected/a8db912c-a862-4940-b690-b85fc3f09c6c-kube-api-access-qfxwp\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:41:32.358729 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.358676    2563 generic.go:356] "Generic (PLEG): container finished" podID="a8db912c-a862-4940-b690-b85fc3f09c6c" containerID="43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4" exitCode=0
Sep 09 21:41:32.358923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.358739    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fdpts"
Sep 09 21:41:32.358923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.358784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fdpts" event={"ID":"a8db912c-a862-4940-b690-b85fc3f09c6c","Type":"ContainerDied","Data":"43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4"}
Sep 09 21:41:32.358923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.358848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fdpts" event={"ID":"a8db912c-a862-4940-b690-b85fc3f09c6c","Type":"ContainerDied","Data":"37f829785f92b768eca63cc8fc0d9f09712c53a02b6b9e243ce58f1f0f648c6c"}
Sep 09 21:41:32.358923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.358869    2563 scope.go:117] "RemoveContainer" containerID="43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4"
Sep 09 21:41:32.368365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.368343    2563 scope.go:117] "RemoveContainer" containerID="43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4"
Sep 09 21:41:32.368676 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:41:32.368654    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4\": container with ID starting with 43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4 not found: ID does not exist" containerID="43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4"
Sep 09 21:41:32.368763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.368685    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4"} err="failed to get container status \"43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4\": rpc error: code = NotFound desc = could not find container \"43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4\": container with ID starting with 43add33bbf831099ce77d24d9b72536942e3bb3dc75e976e7bf40fd91549abc4 not found: ID does not exist"
Sep 09 21:41:32.380903 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.380874    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fdpts"]
Sep 09 21:41:32.384843 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:32.384821    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fdpts"]
Sep 09 21:41:33.045489 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:33.045454    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53d510ac-f4c9-4d26-aadc-7565bdb95873" path="/var/lib/kubelet/pods/53d510ac-f4c9-4d26-aadc-7565bdb95873/volumes"
Sep 09 21:41:33.045683 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:33.045671    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8db912c-a862-4940-b690-b85fc3f09c6c" path="/var/lib/kubelet/pods/a8db912c-a862-4940-b690-b85fc3f09c6c/volumes"
Sep 09 21:41:33.045994 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:41:33.045982    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5ea0ef9-f254-4efd-be06-b2d4927f1b72" path="/var/lib/kubelet/pods/c5ea0ef9-f254-4efd-be06-b2d4927f1b72/volumes"
Sep 09 21:42:12.188546 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.188463    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lswx7"]
Sep 09 21:42:12.189022 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.188828    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8db912c-a862-4940-b690-b85fc3f09c6c" containerName="authorino"
Sep 09 21:42:12.189022 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.188839    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8db912c-a862-4940-b690-b85fc3f09c6c" containerName="authorino"
Sep 09 21:42:12.189022 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.188910    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8db912c-a862-4940-b690-b85fc3f09c6c" containerName="authorino"
Sep 09 21:42:12.191600 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.191580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:12.195016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.194995    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 21:42:12.195301 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.195282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vd9p9\""
Sep 09 21:42:12.203941 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.203918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lswx7"]
Sep 09 21:42:12.263277 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.263234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a06f547-5c0e-452b-8a1d-a9ab4806659c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lswx7\" (UID: \"2a06f547-5c0e-452b-8a1d-a9ab4806659c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:12.263467 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.263402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2pv2\" (UniqueName: \"kubernetes.io/projected/2a06f547-5c0e-452b-8a1d-a9ab4806659c-kube-api-access-j2pv2\") pod \"payload-processing-5b54b7fbb7-lswx7\" (UID: \"2a06f547-5c0e-452b-8a1d-a9ab4806659c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:12.364894 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.364847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2pv2\" (UniqueName: \"kubernetes.io/projected/2a06f547-5c0e-452b-8a1d-a9ab4806659c-kube-api-access-j2pv2\") pod \"payload-processing-5b54b7fbb7-lswx7\" (UID: \"2a06f547-5c0e-452b-8a1d-a9ab4806659c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:12.365071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.364932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a06f547-5c0e-452b-8a1d-a9ab4806659c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lswx7\" (UID: \"2a06f547-5c0e-452b-8a1d-a9ab4806659c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:12.365606 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.365584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a06f547-5c0e-452b-8a1d-a9ab4806659c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lswx7\" (UID: \"2a06f547-5c0e-452b-8a1d-a9ab4806659c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:12.373806 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.373776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2pv2\" (UniqueName: \"kubernetes.io/projected/2a06f547-5c0e-452b-8a1d-a9ab4806659c-kube-api-access-j2pv2\") pod \"payload-processing-5b54b7fbb7-lswx7\" (UID: \"2a06f547-5c0e-452b-8a1d-a9ab4806659c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:12.508854 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.508762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:12.781913 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:12.781866    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lswx7"]
Sep 09 21:42:12.784462 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:42:12.784435    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a06f547_5c0e_452b_8a1d_a9ab4806659c.slice/crio-27dc861e4a846a8959e73c28237467a7790b5e649913cc4c0a228c1c00d13072 WatchSource:0}: Error finding container 27dc861e4a846a8959e73c28237467a7790b5e649913cc4c0a228c1c00d13072: Status 404 returned error can't find the container with id 27dc861e4a846a8959e73c28237467a7790b5e649913cc4c0a228c1c00d13072
Sep 09 21:42:13.544732 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:13.544669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7" event={"ID":"2a06f547-5c0e-452b-8a1d-a9ab4806659c","Type":"ContainerStarted","Data":"27dc861e4a846a8959e73c28237467a7790b5e649913cc4c0a228c1c00d13072"}
Sep 09 21:42:15.554997 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:15.554959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7" event={"ID":"2a06f547-5c0e-452b-8a1d-a9ab4806659c","Type":"ContainerStarted","Data":"abe925f630d6e1c85b5fe682a99935c6a27b29f498f87166ffeb199a0a0c8a8e"}
Sep 09 21:42:15.555376 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:15.555026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:15.571344 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:15.571296    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7" podStartSLOduration=1.809072488 podStartE2EDuration="3.571281337s" totalImagesPullingTime="1.762208849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:15.570381338 +0000 UTC m=+541.056264376" watchObservedRunningTime="2026-09-09 21:42:15.571281337 +0000 UTC m=+541.057164375"
Sep 09 21:42:26.563424 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:26.563390    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lswx7"
Sep 09 21:42:56.279934 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.279896    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"]
Sep 09 21:42:56.284201 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.284179    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.286943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.286920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 21:42:56.288415 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.288392    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wqwhc\""
Sep 09 21:42:56.288531 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.288415    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 21:42:56.288531 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.288392    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 21:42:56.292439 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.292353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"]
Sep 09 21:42:56.355763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.355727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.355942 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.355802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.355942 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.355829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.356016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.355971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qgxn\" (UniqueName: \"kubernetes.io/projected/bcb4187e-31b9-4f52-8292-9e29854fe9d9-kube-api-access-9qgxn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.356100 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.356012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.356100 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.356047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb4187e-31b9-4f52-8292-9e29854fe9d9-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457102 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qgxn\" (UniqueName: \"kubernetes.io/projected/bcb4187e-31b9-4f52-8292-9e29854fe9d9-kube-api-access-9qgxn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457102 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457326 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb4187e-31b9-4f52-8292-9e29854fe9d9-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457326 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457326 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457326 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457546 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457609 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.457664 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.457618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.459591 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.459569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb4187e-31b9-4f52-8292-9e29854fe9d9-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.459907 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.459888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb4187e-31b9-4f52-8292-9e29854fe9d9-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.465462 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.465439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qgxn\" (UniqueName: \"kubernetes.io/projected/bcb4187e-31b9-4f52-8292-9e29854fe9d9-kube-api-access-9qgxn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q\" (UID: \"bcb4187e-31b9-4f52-8292-9e29854fe9d9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.597403 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.597315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:42:56.869350 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:56.869327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"]
Sep 09 21:42:56.871542 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:42:56.871512    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb4187e_31b9_4f52_8292_9e29854fe9d9.slice/crio-ec94f63aab5f53ecbc26a4ac62e6a91000bf36e6aaf1d292b3d92c17a4df1652 WatchSource:0}: Error finding container ec94f63aab5f53ecbc26a4ac62e6a91000bf36e6aaf1d292b3d92c17a4df1652: Status 404 returned error can't find the container with id ec94f63aab5f53ecbc26a4ac62e6a91000bf36e6aaf1d292b3d92c17a4df1652
Sep 09 21:42:57.760684 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:57.760633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q" event={"ID":"bcb4187e-31b9-4f52-8292-9e29854fe9d9","Type":"ContainerStarted","Data":"ec94f63aab5f53ecbc26a4ac62e6a91000bf36e6aaf1d292b3d92c17a4df1652"}
Sep 09 21:42:58.766663 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:58.766621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q" event={"ID":"bcb4187e-31b9-4f52-8292-9e29854fe9d9","Type":"ContainerStarted","Data":"e91fec9e79f286b5f4125002916c7ec150977ea2560183619e6762890a6550c0"}
Sep 09 21:42:58.783901 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:42:58.783851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q" podStartSLOduration=1.693248294 podStartE2EDuration="2.783835388s" totalImagesPullingTime="1.090587094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:58.782991613 +0000 UTC m=+584.268874653" watchObservedRunningTime="2026-09-09 21:42:58.783835388 +0000 UTC m=+584.269718426"
Sep 09 21:43:06.597466 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:06.597425    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:43:06.597971 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:06.597497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:43:06.610103 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:06.610074    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:43:06.816619 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:06.816589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q"
Sep 09 21:43:17.991706 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:17.991664    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"]
Sep 09 21:43:17.995265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:17.995241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:17.997829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:17.997805    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 21:43:18.005375 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.005349    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"]
Sep 09 21:43:18.155284 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.155241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.155468 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.155291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.155468 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.155409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knpn\" (UniqueName: \"kubernetes.io/projected/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-kube-api-access-6knpn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.155468 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.155460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.155605 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.155486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.155605 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.155512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256309 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knpn\" (UniqueName: \"kubernetes.io/projected/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-kube-api-access-6knpn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256309 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256309 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256576 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256576 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256576 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256762 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256762 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.256886 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.256847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.258863 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.258842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.259033 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.259014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.264255 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.264231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knpn\" (UniqueName: \"kubernetes.io/projected/41ea82f9-88ef-4d5d-8c6b-9fa36e575528-kube-api-access-6knpn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rjdf\" (UID: \"41ea82f9-88ef-4d5d-8c6b-9fa36e575528\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.308367 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.308312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:18.579366 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.579341    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"]
Sep 09 21:43:18.581201 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:43:18.581173    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ea82f9_88ef_4d5d_8c6b_9fa36e575528.slice/crio-0cdd22cf613b8f3f34d808ed37853b9507eafa8e0d32c06646d621dd74e30559 WatchSource:0}: Error finding container 0cdd22cf613b8f3f34d808ed37853b9507eafa8e0d32c06646d621dd74e30559: Status 404 returned error can't find the container with id 0cdd22cf613b8f3f34d808ed37853b9507eafa8e0d32c06646d621dd74e30559
Sep 09 21:43:18.866790 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.866754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf" event={"ID":"41ea82f9-88ef-4d5d-8c6b-9fa36e575528","Type":"ContainerStarted","Data":"eff5fe03018d803f2ec493401876711331aafd39c5059f3bd2649a2bcb703098"}
Sep 09 21:43:18.866960 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.866831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf" event={"ID":"41ea82f9-88ef-4d5d-8c6b-9fa36e575528","Type":"ContainerStarted","Data":"0cdd22cf613b8f3f34d808ed37853b9507eafa8e0d32c06646d621dd74e30559"}
Sep 09 21:43:18.884329 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:18.884274    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf" podStartSLOduration=1.70058814 podStartE2EDuration="1.88425899s" totalImagesPullingTime="183.67085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:18.882159791 +0000 UTC m=+604.368042828" watchObservedRunningTime="2026-09-09 21:43:18.88425899 +0000 UTC m=+604.370142027"
Sep 09 21:43:28.308470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:28.308424    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:28.311149 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:28.308496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:28.321035 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:28.321017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:28.926187 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:28.926146    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rjdf"
Sep 09 21:43:30.194291 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.194254    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"]
Sep 09 21:43:30.196962 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.196940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.199555 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.199532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 21:43:30.207013 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.206987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"]
Sep 09 21:43:30.256278 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.256229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.256494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.256320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.256494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.256438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgdf\" (UniqueName: \"kubernetes.io/projected/2e95c85d-142e-4bbb-8618-763259a9a0bf-kube-api-access-6fgdf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.256494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.256483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.256643 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.256568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.256643 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.256635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e95c85d-142e-4bbb-8618-763259a9a0bf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357258 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e95c85d-142e-4bbb-8618-763259a9a0bf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357450 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357450 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357450 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgdf\" (UniqueName: \"kubernetes.io/projected/2e95c85d-142e-4bbb-8618-763259a9a0bf-kube-api-access-6fgdf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357623 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357623 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357831 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.357949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.357930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.359702 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.359682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e95c85d-142e-4bbb-8618-763259a9a0bf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.359934 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.359916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e95c85d-142e-4bbb-8618-763259a9a0bf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.365454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.365430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgdf\" (UniqueName: \"kubernetes.io/projected/2e95c85d-142e-4bbb-8618-763259a9a0bf-kube-api-access-6fgdf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l\" (UID: \"2e95c85d-142e-4bbb-8618-763259a9a0bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.509285 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.509185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:30.777145 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.777117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"]
Sep 09 21:43:30.779066 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:43:30.779038    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e95c85d_142e_4bbb_8618_763259a9a0bf.slice/crio-c827ee8d1177a4415c2dabdb3c070d5f76217c87403550cb4318f2304a25cc71 WatchSource:0}: Error finding container c827ee8d1177a4415c2dabdb3c070d5f76217c87403550cb4318f2304a25cc71: Status 404 returned error can't find the container with id c827ee8d1177a4415c2dabdb3c070d5f76217c87403550cb4318f2304a25cc71
Sep 09 21:43:30.780355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.780340    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:43:30.925765 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:30.925709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l" event={"ID":"2e95c85d-142e-4bbb-8618-763259a9a0bf","Type":"ContainerStarted","Data":"c827ee8d1177a4415c2dabdb3c070d5f76217c87403550cb4318f2304a25cc71"}
Sep 09 21:43:31.932820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:31.932785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l" event={"ID":"2e95c85d-142e-4bbb-8618-763259a9a0bf","Type":"ContainerStarted","Data":"b36b4047f13f1852a77426281d1514f49627bd20176267a135a0a0227281e80c"}
Sep 09 21:43:31.951055 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:31.950997    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l" podStartSLOduration=1.778424547 podStartE2EDuration="1.950981126s" totalImagesPullingTime="172.556579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:31.948970749 +0000 UTC m=+617.434853788" watchObservedRunningTime="2026-09-09 21:43:31.950981126 +0000 UTC m=+617.436864163"
Sep 09 21:43:40.510049 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:40.509953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:40.510049 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:40.510032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:40.522186 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:40.522157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:43:40.991201 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:43:40.991165    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l"
Sep 09 21:44:05.894353 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.894323    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"]
Sep 09 21:44:05.896954 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.896937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:05.899503 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.899480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 21:44:05.906218 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.906191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"]
Sep 09 21:44:05.986768 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.986711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee25eb0-85cb-4f46-b0cf-b76b88743263-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:05.986943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.986852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:05.986943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.986887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:05.986943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.986908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:05.986943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.986925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:05.987105 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:05.986994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89sxz\" (UniqueName: \"kubernetes.io/projected/1ee25eb0-85cb-4f46-b0cf-b76b88743263-kube-api-access-89sxz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.088467 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.088425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee25eb0-85cb-4f46-b0cf-b76b88743263-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.088668 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.088514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.088668 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.088538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.088668 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.088560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.088668 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.088580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.088668 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.088607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89sxz\" (UniqueName: \"kubernetes.io/projected/1ee25eb0-85cb-4f46-b0cf-b76b88743263-kube-api-access-89sxz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.089047 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.089015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.089047 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.089039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.089173 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.089095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.091040 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.091011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee25eb0-85cb-4f46-b0cf-b76b88743263-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.091191 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.091172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee25eb0-85cb-4f46-b0cf-b76b88743263-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.097801 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.097778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89sxz\" (UniqueName: \"kubernetes.io/projected/1ee25eb0-85cb-4f46-b0cf-b76b88743263-kube-api-access-89sxz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k\" (UID: \"1ee25eb0-85cb-4f46-b0cf-b76b88743263\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.207082 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.206996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:06.479438 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:06.479408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"]
Sep 09 21:44:06.480945 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:44:06.480924    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ee25eb0_85cb_4f46_b0cf_b76b88743263.slice/crio-1a3e62fdb65b9405c4c4ea73d4982597c94f94ea36190b615443b5768d91bc51 WatchSource:0}: Error finding container 1a3e62fdb65b9405c4c4ea73d4982597c94f94ea36190b615443b5768d91bc51: Status 404 returned error can't find the container with id 1a3e62fdb65b9405c4c4ea73d4982597c94f94ea36190b615443b5768d91bc51
Sep 09 21:44:07.104455 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:07.104417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k" event={"ID":"1ee25eb0-85cb-4f46-b0cf-b76b88743263","Type":"ContainerStarted","Data":"7a143b6663e6dfe519846f6e2d69342b18eb8b66c06e5ae311e921a5d1315c20"}
Sep 09 21:44:07.104455 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:07.104458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k" event={"ID":"1ee25eb0-85cb-4f46-b0cf-b76b88743263","Type":"ContainerStarted","Data":"1a3e62fdb65b9405c4c4ea73d4982597c94f94ea36190b615443b5768d91bc51"}
Sep 09 21:44:07.123347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:07.123287    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k" podStartSLOduration=1.9378872010000001 podStartE2EDuration="2.123271783s" totalImagesPullingTime="185.384582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:44:07.121694277 +0000 UTC m=+652.607577317" watchObservedRunningTime="2026-09-09 21:44:07.123271783 +0000 UTC m=+652.609154846"
Sep 09 21:44:16.207462 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:16.207418    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:16.207462 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:16.207472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:16.219953 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:16.219926    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:44:17.161393 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:44:17.161362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k"
Sep 09 21:45:27.406741 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.406635    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-658ddbb688-wdqh9"]
Sep 09 21:45:27.408910 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.408890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-658ddbb688-wdqh9"
Sep 09 21:45:27.418229 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.418206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-658ddbb688-wdqh9"]
Sep 09 21:45:27.557996 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.557969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61297c74-5dc5-4627-ae8d-263c39bdd2d0-tls-cert\") pod \"authorino-658ddbb688-wdqh9\" (UID: \"61297c74-5dc5-4627-ae8d-263c39bdd2d0\") " pod="kuadrant-system/authorino-658ddbb688-wdqh9"
Sep 09 21:45:27.558156 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.558011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sppq\" (UniqueName: \"kubernetes.io/projected/61297c74-5dc5-4627-ae8d-263c39bdd2d0-kube-api-access-9sppq\") pod \"authorino-658ddbb688-wdqh9\" (UID: \"61297c74-5dc5-4627-ae8d-263c39bdd2d0\") " pod="kuadrant-system/authorino-658ddbb688-wdqh9"
Sep 09 21:45:27.659448 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.659342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61297c74-5dc5-4627-ae8d-263c39bdd2d0-tls-cert\") pod \"authorino-658ddbb688-wdqh9\" (UID: \"61297c74-5dc5-4627-ae8d-263c39bdd2d0\") " pod="kuadrant-system/authorino-658ddbb688-wdqh9"
Sep 09 21:45:27.659448 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.659411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sppq\" (UniqueName: \"kubernetes.io/projected/61297c74-5dc5-4627-ae8d-263c39bdd2d0-kube-api-access-9sppq\") pod \"authorino-658ddbb688-wdqh9\" (UID: \"61297c74-5dc5-4627-ae8d-263c39bdd2d0\") " pod="kuadrant-system/authorino-658ddbb688-wdqh9"
Sep 09 21:45:27.662022 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.661990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61297c74-5dc5-4627-ae8d-263c39bdd2d0-tls-cert\") pod \"authorino-658ddbb688-wdqh9\" (UID: \"61297c74-5dc5-4627-ae8d-263c39bdd2d0\") " pod="kuadrant-system/authorino-658ddbb688-wdqh9"
Sep 09 21:45:27.667428 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.667403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sppq\" (UniqueName: \"kubernetes.io/projected/61297c74-5dc5-4627-ae8d-263c39bdd2d0-kube-api-access-9sppq\") pod \"authorino-658ddbb688-wdqh9\" (UID: \"61297c74-5dc5-4627-ae8d-263c39bdd2d0\") " pod="kuadrant-system/authorino-658ddbb688-wdqh9"
Sep 09 21:45:27.718303 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.718259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-658ddbb688-wdqh9"
Sep 09 21:45:27.987856 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:27.987831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-658ddbb688-wdqh9"]
Sep 09 21:45:27.989915 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:45:27.989877    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61297c74_5dc5_4627_ae8d_263c39bdd2d0.slice/crio-1de8e66b1be31825fb6b7c373a61ef5f064f9184eb0b140c1516c9ef9a6b4653 WatchSource:0}: Error finding container 1de8e66b1be31825fb6b7c373a61ef5f064f9184eb0b140c1516c9ef9a6b4653: Status 404 returned error can't find the container with id 1de8e66b1be31825fb6b7c373a61ef5f064f9184eb0b140c1516c9ef9a6b4653
Sep 09 21:45:28.494860 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:28.494824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-658ddbb688-wdqh9" event={"ID":"61297c74-5dc5-4627-ae8d-263c39bdd2d0","Type":"ContainerStarted","Data":"1de8e66b1be31825fb6b7c373a61ef5f064f9184eb0b140c1516c9ef9a6b4653"}
Sep 09 21:45:29.502117 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.502075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-658ddbb688-wdqh9" event={"ID":"61297c74-5dc5-4627-ae8d-263c39bdd2d0","Type":"ContainerStarted","Data":"5f7286bf5ad0b39085eb174f6facf966c9b8eb9a1f1b757877c606316561d61a"}
Sep 09 21:45:29.519175 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.519114    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-658ddbb688-wdqh9" podStartSLOduration=1.743675004 podStartE2EDuration="2.519093631s" totalImagesPullingTime="775.418627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:29.517840413 +0000 UTC m=+735.003723451" watchObservedRunningTime="2026-09-09 21:45:29.519093631 +0000 UTC m=+735.004976673"
Sep 09 21:45:29.548331 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.548290    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54bb746d97-8ghgr"]
Sep 09 21:45:29.548555 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.548526    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-54bb746d97-8ghgr" podUID="3d2d553d-0073-4423-a2e0-017791334ac4" containerName="authorino" containerID="cri-o://c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb" gracePeriod=30
Sep 09 21:45:29.787790 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.787753    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:45:29.881794 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.881763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d2d553d-0073-4423-a2e0-017791334ac4-tls-cert\") pod \"3d2d553d-0073-4423-a2e0-017791334ac4\" (UID: \"3d2d553d-0073-4423-a2e0-017791334ac4\") "
Sep 09 21:45:29.881794 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.881801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7pz2\" (UniqueName: \"kubernetes.io/projected/3d2d553d-0073-4423-a2e0-017791334ac4-kube-api-access-p7pz2\") pod \"3d2d553d-0073-4423-a2e0-017791334ac4\" (UID: \"3d2d553d-0073-4423-a2e0-017791334ac4\") "
Sep 09 21:45:29.884099 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.884070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2d553d-0073-4423-a2e0-017791334ac4-kube-api-access-p7pz2" (OuterVolumeSpecName: "kube-api-access-p7pz2") pod "3d2d553d-0073-4423-a2e0-017791334ac4" (UID: "3d2d553d-0073-4423-a2e0-017791334ac4"). InnerVolumeSpecName "kube-api-access-p7pz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:45:29.893340 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.893306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2d553d-0073-4423-a2e0-017791334ac4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3d2d553d-0073-4423-a2e0-017791334ac4" (UID: "3d2d553d-0073-4423-a2e0-017791334ac4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:45:29.983550 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.983509    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d2d553d-0073-4423-a2e0-017791334ac4-tls-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:45:29.989732 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:29.983553    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7pz2\" (UniqueName: \"kubernetes.io/projected/3d2d553d-0073-4423-a2e0-017791334ac4-kube-api-access-p7pz2\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:45:30.508567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.508531    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d2d553d-0073-4423-a2e0-017791334ac4" containerID="c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb" exitCode=0
Sep 09 21:45:30.509066 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.508597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54bb746d97-8ghgr"
Sep 09 21:45:30.509066 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.508619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54bb746d97-8ghgr" event={"ID":"3d2d553d-0073-4423-a2e0-017791334ac4","Type":"ContainerDied","Data":"c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb"}
Sep 09 21:45:30.509066 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.508670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54bb746d97-8ghgr" event={"ID":"3d2d553d-0073-4423-a2e0-017791334ac4","Type":"ContainerDied","Data":"658f3cbc69e0050686e7b1b107669c9ef9e9d176e126ffbe875be1c86510c344"}
Sep 09 21:45:30.509066 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.508690    2563 scope.go:117] "RemoveContainer" containerID="c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb"
Sep 09 21:45:30.520245 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.520222    2563 scope.go:117] "RemoveContainer" containerID="c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb"
Sep 09 21:45:30.520587 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:45:30.520566    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb\": container with ID starting with c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb not found: ID does not exist" containerID="c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb"
Sep 09 21:45:30.520667 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.520597    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb"} err="failed to get container status \"c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb\": rpc error: code = NotFound desc = could not find container \"c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb\": container with ID starting with c220f673bbef5d73412b0b37a5487b6379c140a8b1e902e08d7856891d02b2bb not found: ID does not exist"
Sep 09 21:45:30.533652 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.533608    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54bb746d97-8ghgr"]
Sep 09 21:45:30.537553 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:30.537527    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54bb746d97-8ghgr"]
Sep 09 21:45:31.048650 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:31.048618    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2d553d-0073-4423-a2e0-017791334ac4" path="/var/lib/kubelet/pods/3d2d553d-0073-4423-a2e0-017791334ac4/volumes"
Sep 09 21:45:52.672516 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.672473    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"]
Sep 09 21:45:52.673001 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.672986    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2d553d-0073-4423-a2e0-017791334ac4" containerName="authorino"
Sep 09 21:45:52.673049 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.673003    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2d553d-0073-4423-a2e0-017791334ac4" containerName="authorino"
Sep 09 21:45:52.673080 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.673067    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2d553d-0073-4423-a2e0-017791334ac4" containerName="authorino"
Sep 09 21:45:52.675132 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.675117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:52.696234 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.696207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-390abc-dockercfg-8qdq7\""
Sep 09 21:45:52.696551 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.696536    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-390abc\""
Sep 09 21:45:52.714768 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.714735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"]
Sep 09 21:45:52.814214 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.814175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff02b24-2e79-461a-a09c-1d267306d5a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4\" (UID: \"6ff02b24-2e79-461a-a09c-1d267306d5a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:52.814418 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.814331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88bj5\" (UniqueName: \"kubernetes.io/projected/6ff02b24-2e79-461a-a09c-1d267306d5a6-kube-api-access-88bj5\") pod \"payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4\" (UID: \"6ff02b24-2e79-461a-a09c-1d267306d5a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:52.915090 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.915049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bj5\" (UniqueName: \"kubernetes.io/projected/6ff02b24-2e79-461a-a09c-1d267306d5a6-kube-api-access-88bj5\") pod \"payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4\" (UID: \"6ff02b24-2e79-461a-a09c-1d267306d5a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:52.915263 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.915103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff02b24-2e79-461a-a09c-1d267306d5a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4\" (UID: \"6ff02b24-2e79-461a-a09c-1d267306d5a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:52.915897 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.915873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff02b24-2e79-461a-a09c-1d267306d5a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4\" (UID: \"6ff02b24-2e79-461a-a09c-1d267306d5a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:52.929703 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.929644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88bj5\" (UniqueName: \"kubernetes.io/projected/6ff02b24-2e79-461a-a09c-1d267306d5a6-kube-api-access-88bj5\") pod \"payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4\" (UID: \"6ff02b24-2e79-461a-a09c-1d267306d5a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:52.984633 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:52.984580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:53.261145 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.261118    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"]
Sep 09 21:45:53.263123 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:45:53.263094    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff02b24_2e79_461a_a09c_1d267306d5a6.slice/crio-28bd8a188c857fa4b6ca84c21e516ebd2086298deb27d1a5e5ce566e3178b634 WatchSource:0}: Error finding container 28bd8a188c857fa4b6ca84c21e516ebd2086298deb27d1a5e5ce566e3178b634: Status 404 returned error can't find the container with id 28bd8a188c857fa4b6ca84c21e516ebd2086298deb27d1a5e5ce566e3178b634
Sep 09 21:45:53.621075 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.621036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4" event={"ID":"6ff02b24-2e79-461a-a09c-1d267306d5a6","Type":"ContainerStarted","Data":"965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93"}
Sep 09 21:45:53.621256 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.621103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4" event={"ID":"6ff02b24-2e79-461a-a09c-1d267306d5a6","Type":"ContainerStarted","Data":"28bd8a188c857fa4b6ca84c21e516ebd2086298deb27d1a5e5ce566e3178b634"}
Sep 09 21:45:53.621256 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.621132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:45:53.637796 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.637691    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4" podStartSLOduration=1.6376729490000002 podStartE2EDuration="1.637672949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:53.636769007 +0000 UTC m=+759.122652046" watchObservedRunningTime="2026-09-09 21:45:53.637672949 +0000 UTC m=+759.123555986"
Sep 09 21:45:53.700524 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.700487    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"]
Sep 09 21:45:53.703061 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.703039    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:53.705606 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.705578    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2aaab1\""
Sep 09 21:45:53.715545 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.715520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"]
Sep 09 21:45:53.750425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.750392    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"]
Sep 09 21:45:53.753185 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.753168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:53.756017 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.755993    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2aaab1-dockercfg-t8nrx\""
Sep 09 21:45:53.770228 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.770203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"]
Sep 09 21:45:53.823962 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.823929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tclzw\" (UniqueName: \"kubernetes.io/projected/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-kube-api-access-tclzw\") pod \"payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g\" (UID: \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:53.823962 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.823970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g\" (UID: \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:53.925411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.925323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tclzw\" (UniqueName: \"kubernetes.io/projected/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-kube-api-access-tclzw\") pod \"payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g\" (UID: \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:53.925411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.925366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g\" (UID: \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:53.925411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.925391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nrs\" (UniqueName: \"kubernetes.io/projected/ff732478-a16b-4998-b61a-113bd9cc83a0-kube-api-access-z4nrs\") pod \"payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg\" (UID: \"ff732478-a16b-4998-b61a-113bd9cc83a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:53.925632 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.925435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff732478-a16b-4998-b61a-113bd9cc83a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg\" (UID: \"ff732478-a16b-4998-b61a-113bd9cc83a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:53.926126 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.926107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g\" (UID: \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:53.933636 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:53.933613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tclzw\" (UniqueName: \"kubernetes.io/projected/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-kube-api-access-tclzw\") pod \"payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g\" (UID: \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:54.015912 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.015873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:54.026256 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.026222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff732478-a16b-4998-b61a-113bd9cc83a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg\" (UID: \"ff732478-a16b-4998-b61a-113bd9cc83a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:54.026381 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.026356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nrs\" (UniqueName: \"kubernetes.io/projected/ff732478-a16b-4998-b61a-113bd9cc83a0-kube-api-access-z4nrs\") pod \"payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg\" (UID: \"ff732478-a16b-4998-b61a-113bd9cc83a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:54.026991 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.026969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff732478-a16b-4998-b61a-113bd9cc83a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg\" (UID: \"ff732478-a16b-4998-b61a-113bd9cc83a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:54.034800 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.034772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nrs\" (UniqueName: \"kubernetes.io/projected/ff732478-a16b-4998-b61a-113bd9cc83a0-kube-api-access-z4nrs\") pod \"payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg\" (UID: \"ff732478-a16b-4998-b61a-113bd9cc83a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:54.064477 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.064440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:54.286547 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.286516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"]
Sep 09 21:45:54.288084 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:45:54.288058    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9ab66b_bef8_4d34_b0d3_f7124d2e395c.slice/crio-d526f47efe0ad75718bfc95e238cf556313d7291b007b0f78facaa6fd7c2def6 WatchSource:0}: Error finding container d526f47efe0ad75718bfc95e238cf556313d7291b007b0f78facaa6fd7c2def6: Status 404 returned error can't find the container with id d526f47efe0ad75718bfc95e238cf556313d7291b007b0f78facaa6fd7c2def6
Sep 09 21:45:54.345734 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.345692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"]
Sep 09 21:45:54.347643 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:45:54.347585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff732478_a16b_4998_b61a_113bd9cc83a0.slice/crio-b8ad5f108fd7f48e13be23f198e1ef97e408917c7a7adfc7fbfc677da0f19484 WatchSource:0}: Error finding container b8ad5f108fd7f48e13be23f198e1ef97e408917c7a7adfc7fbfc677da0f19484: Status 404 returned error can't find the container with id b8ad5f108fd7f48e13be23f198e1ef97e408917c7a7adfc7fbfc677da0f19484
Sep 09 21:45:54.628768 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.628732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg" event={"ID":"ff732478-a16b-4998-b61a-113bd9cc83a0","Type":"ContainerStarted","Data":"a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b"}
Sep 09 21:45:54.628976 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.628795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg" event={"ID":"ff732478-a16b-4998-b61a-113bd9cc83a0","Type":"ContainerStarted","Data":"b8ad5f108fd7f48e13be23f198e1ef97e408917c7a7adfc7fbfc677da0f19484"}
Sep 09 21:45:54.629032 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.628999    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:45:54.630788 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.630759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g" event={"ID":"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c","Type":"ContainerStarted","Data":"b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724"}
Sep 09 21:45:54.630928 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.630813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g" event={"ID":"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c","Type":"ContainerStarted","Data":"d526f47efe0ad75718bfc95e238cf556313d7291b007b0f78facaa6fd7c2def6"}
Sep 09 21:45:54.631000 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.630984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:45:54.645797 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.645740    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg" podStartSLOduration=1.6456972300000001 podStartE2EDuration="1.64569723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:54.645197945 +0000 UTC m=+760.131080986" watchObservedRunningTime="2026-09-09 21:45:54.64569723 +0000 UTC m=+760.131580251"
Sep 09 21:45:54.661771 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:45:54.661709    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g" podStartSLOduration=1.661694003 podStartE2EDuration="1.661694003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:54.660138112 +0000 UTC m=+760.146021150" watchObservedRunningTime="2026-09-09 21:45:54.661694003 +0000 UTC m=+760.147577042"
Sep 09 21:46:01.407356 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.407322    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"]
Sep 09 21:46:01.410322 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.410302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:01.413484 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.413453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-04d63732-dockercfg-nn759\""
Sep 09 21:46:01.413484 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.413471    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-04d63732\""
Sep 09 21:46:01.422418 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.422387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"]
Sep 09 21:46:01.439954 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.439923    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"]
Sep 09 21:46:01.442526 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.442508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:01.456578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.456553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"]
Sep 09 21:46:01.599665 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.599623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8437a2a-16ac-4c0d-8632-bd08177fd537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq\" (UID: \"d8437a2a-16ac-4c0d-8632-bd08177fd537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:01.599890 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.599677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhr4\" (UniqueName: \"kubernetes.io/projected/d8437a2a-16ac-4c0d-8632-bd08177fd537-kube-api-access-kbhr4\") pod \"payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq\" (UID: \"d8437a2a-16ac-4c0d-8632-bd08177fd537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:01.599890 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.599744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6332fd8b-3630-4baa-bad2-b647bbc517de-plugins-config-volume\") pod \"payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm\" (UID: \"6332fd8b-3630-4baa-bad2-b647bbc517de\") " pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:01.599890 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.599780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6mc\" (UniqueName: \"kubernetes.io/projected/6332fd8b-3630-4baa-bad2-b647bbc517de-kube-api-access-tb6mc\") pod \"payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm\" (UID: \"6332fd8b-3630-4baa-bad2-b647bbc517de\") " pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:01.700514 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.700397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhr4\" (UniqueName: \"kubernetes.io/projected/d8437a2a-16ac-4c0d-8632-bd08177fd537-kube-api-access-kbhr4\") pod \"payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq\" (UID: \"d8437a2a-16ac-4c0d-8632-bd08177fd537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:01.700514 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.700463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6332fd8b-3630-4baa-bad2-b647bbc517de-plugins-config-volume\") pod \"payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm\" (UID: \"6332fd8b-3630-4baa-bad2-b647bbc517de\") " pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:01.700514 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.700486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6mc\" (UniqueName: \"kubernetes.io/projected/6332fd8b-3630-4baa-bad2-b647bbc517de-kube-api-access-tb6mc\") pod \"payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm\" (UID: \"6332fd8b-3630-4baa-bad2-b647bbc517de\") " pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:01.700854 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.700572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8437a2a-16ac-4c0d-8632-bd08177fd537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq\" (UID: \"d8437a2a-16ac-4c0d-8632-bd08177fd537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:01.701229 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.701209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8437a2a-16ac-4c0d-8632-bd08177fd537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq\" (UID: \"d8437a2a-16ac-4c0d-8632-bd08177fd537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:01.701265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.701221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6332fd8b-3630-4baa-bad2-b647bbc517de-plugins-config-volume\") pod \"payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm\" (UID: \"6332fd8b-3630-4baa-bad2-b647bbc517de\") " pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:01.708526 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.708494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6mc\" (UniqueName: \"kubernetes.io/projected/6332fd8b-3630-4baa-bad2-b647bbc517de-kube-api-access-tb6mc\") pod \"payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm\" (UID: \"6332fd8b-3630-4baa-bad2-b647bbc517de\") " pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:01.708887 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.708870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhr4\" (UniqueName: \"kubernetes.io/projected/d8437a2a-16ac-4c0d-8632-bd08177fd537-kube-api-access-kbhr4\") pod \"payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq\" (UID: \"d8437a2a-16ac-4c0d-8632-bd08177fd537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:01.729773 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.729743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:01.752157 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:01.752123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:02.003184 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.003157    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"]
Sep 09 21:46:02.004631 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:46:02.004608    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8437a2a_16ac_4c0d_8632_bd08177fd537.slice/crio-a7f81e19bcd4abc58bb71b5ac584c1cd7a4341aa54cbd7692a99668ba0fb27c0 WatchSource:0}: Error finding container a7f81e19bcd4abc58bb71b5ac584c1cd7a4341aa54cbd7692a99668ba0fb27c0: Status 404 returned error can't find the container with id a7f81e19bcd4abc58bb71b5ac584c1cd7a4341aa54cbd7692a99668ba0fb27c0
Sep 09 21:46:02.034933 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.034911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"]
Sep 09 21:46:02.038932 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:46:02.038902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6332fd8b_3630_4baa_bad2_b647bbc517de.slice/crio-d1c965fb9630ab7900e93337500743f378efabf0554587d7ab52b1b0126df544 WatchSource:0}: Error finding container d1c965fb9630ab7900e93337500743f378efabf0554587d7ab52b1b0126df544: Status 404 returned error can't find the container with id d1c965fb9630ab7900e93337500743f378efabf0554587d7ab52b1b0126df544
Sep 09 21:46:02.675254 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.675218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq" event={"ID":"d8437a2a-16ac-4c0d-8632-bd08177fd537","Type":"ContainerStarted","Data":"a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2"}
Sep 09 21:46:02.675744 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.675283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq" event={"ID":"d8437a2a-16ac-4c0d-8632-bd08177fd537","Type":"ContainerStarted","Data":"a7f81e19bcd4abc58bb71b5ac584c1cd7a4341aa54cbd7692a99668ba0fb27c0"}
Sep 09 21:46:02.675744 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.675313    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:02.676767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.676708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm" event={"ID":"6332fd8b-3630-4baa-bad2-b647bbc517de","Type":"ContainerStarted","Data":"7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb"}
Sep 09 21:46:02.676895 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.676781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm" event={"ID":"6332fd8b-3630-4baa-bad2-b647bbc517de","Type":"ContainerStarted","Data":"d1c965fb9630ab7900e93337500743f378efabf0554587d7ab52b1b0126df544"}
Sep 09 21:46:02.676895 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.676809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:02.692988 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.692941    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq" podStartSLOduration=1.692928064 podStartE2EDuration="1.692928064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:02.691246223 +0000 UTC m=+768.177129264" watchObservedRunningTime="2026-09-09 21:46:02.692928064 +0000 UTC m=+768.178811101"
Sep 09 21:46:02.707386 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:02.707338    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm" podStartSLOduration=1.707325625 podStartE2EDuration="1.707325625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:02.706355346 +0000 UTC m=+768.192238384" watchObservedRunningTime="2026-09-09 21:46:02.707325625 +0000 UTC m=+768.193208686"
Sep 09 21:46:04.632659 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:04.632632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:46:05.638069 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:05.638034    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:46:05.638513 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:05.638125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:46:13.684355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:13.684318    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:13.684763 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:13.684505    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:24.862893 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.862854    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"]
Sep 09 21:46:24.867707 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.867685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"
Sep 09 21:46:24.870341 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.870316    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:46:24.873133 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.873103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"]
Sep 09 21:46:24.914315 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.914281    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"]
Sep 09 21:46:24.918080 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.918060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:24.920623 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.920596    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-da0a18\""
Sep 09 21:46:24.920751 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.920644    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-da0a18-dockercfg-m899p\""
Sep 09 21:46:24.927776 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.927751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"]
Sep 09 21:46:24.936538 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.936515    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"]
Sep 09 21:46:24.939980 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.939962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:24.949776 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:24.949753    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"]
Sep 09 21:46:25.031731 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.031686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjdl\" (UniqueName: \"kubernetes.io/projected/afc84ea3-4140-4e93-8dbe-01eec364d55f-kube-api-access-cqjdl\") pod \"payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl\" (UID: \"afc84ea3-4140-4e93-8dbe-01eec364d55f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:25.031923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.031865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc84ea3-4140-4e93-8dbe-01eec364d55f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl\" (UID: \"afc84ea3-4140-4e93-8dbe-01eec364d55f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:25.031923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.031902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04d63732-lz88t\" (UID: \"9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"
Sep 09 21:46:25.032044 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.031939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqr2s\" (UniqueName: \"kubernetes.io/projected/33afc968-7590-4088-b21b-73b69e6008fc-kube-api-access-jqr2s\") pod \"payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw\" (UID: \"33afc968-7590-4088-b21b-73b69e6008fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:25.032044 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.031997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33afc968-7590-4088-b21b-73b69e6008fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw\" (UID: \"33afc968-7590-4088-b21b-73b69e6008fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:25.132624 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.132524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc84ea3-4140-4e93-8dbe-01eec364d55f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl\" (UID: \"afc84ea3-4140-4e93-8dbe-01eec364d55f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:25.132624 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.132560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04d63732-lz88t\" (UID: \"9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"
Sep 09 21:46:25.132624 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.132582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqr2s\" (UniqueName: \"kubernetes.io/projected/33afc968-7590-4088-b21b-73b69e6008fc-kube-api-access-jqr2s\") pod \"payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw\" (UID: \"33afc968-7590-4088-b21b-73b69e6008fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:25.132959 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.132708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33afc968-7590-4088-b21b-73b69e6008fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw\" (UID: \"33afc968-7590-4088-b21b-73b69e6008fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:25.132959 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.132786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjdl\" (UniqueName: \"kubernetes.io/projected/afc84ea3-4140-4e93-8dbe-01eec364d55f-kube-api-access-cqjdl\") pod \"payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl\" (UID: \"afc84ea3-4140-4e93-8dbe-01eec364d55f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:25.133334 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.133313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-04d63732-lz88t\" (UID: \"9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"
Sep 09 21:46:25.133426 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.133390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33afc968-7590-4088-b21b-73b69e6008fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw\" (UID: \"33afc968-7590-4088-b21b-73b69e6008fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:25.133481 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.133417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc84ea3-4140-4e93-8dbe-01eec364d55f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl\" (UID: \"afc84ea3-4140-4e93-8dbe-01eec364d55f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:25.142919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.142895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqr2s\" (UniqueName: \"kubernetes.io/projected/33afc968-7590-4088-b21b-73b69e6008fc-kube-api-access-jqr2s\") pod \"payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw\" (UID: \"33afc968-7590-4088-b21b-73b69e6008fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:25.143027 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.142947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjdl\" (UniqueName: \"kubernetes.io/projected/afc84ea3-4140-4e93-8dbe-01eec364d55f-kube-api-access-cqjdl\") pod \"payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl\" (UID: \"afc84ea3-4140-4e93-8dbe-01eec364d55f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:25.179363 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.179335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"
Sep 09 21:46:25.230274 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.230185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:25.251296 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.251247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:25.456071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.456046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"]
Sep 09 21:46:25.457779 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:46:25.457752    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcfbdc9_ceee_45b9_8a8c_6078b10c8fba.slice/crio-26a58a702f747a7c97f4b7d5bbf9cbfa0355b86ab3cd985fe974b66f15eb190b WatchSource:0}: Error finding container 26a58a702f747a7c97f4b7d5bbf9cbfa0355b86ab3cd985fe974b66f15eb190b: Status 404 returned error can't find the container with id 26a58a702f747a7c97f4b7d5bbf9cbfa0355b86ab3cd985fe974b66f15eb190b
Sep 09 21:46:25.512273 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.512248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"]
Sep 09 21:46:25.513798 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:46:25.513768    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33afc968_7590_4088_b21b_73b69e6008fc.slice/crio-b8284c38ab7a1ab3634f767ae20bb0be403c1cc055d1de5a5f7454d217e9b9fc WatchSource:0}: Error finding container b8284c38ab7a1ab3634f767ae20bb0be403c1cc055d1de5a5f7454d217e9b9fc: Status 404 returned error can't find the container with id b8284c38ab7a1ab3634f767ae20bb0be403c1cc055d1de5a5f7454d217e9b9fc
Sep 09 21:46:25.537360 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.537331    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"]
Sep 09 21:46:25.546902 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:46:25.546870    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc84ea3_4140_4e93_8dbe_01eec364d55f.slice/crio-c6553aff08291d2480259fa06875386db36c46f86b72d39b0f1e44257a07283d WatchSource:0}: Error finding container c6553aff08291d2480259fa06875386db36c46f86b72d39b0f1e44257a07283d: Status 404 returned error can't find the container with id c6553aff08291d2480259fa06875386db36c46f86b72d39b0f1e44257a07283d
Sep 09 21:46:25.798680 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.798579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl" event={"ID":"afc84ea3-4140-4e93-8dbe-01eec364d55f","Type":"ContainerStarted","Data":"1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83"}
Sep 09 21:46:25.798680 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.798646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl" event={"ID":"afc84ea3-4140-4e93-8dbe-01eec364d55f","Type":"ContainerStarted","Data":"c6553aff08291d2480259fa06875386db36c46f86b72d39b0f1e44257a07283d"}
Sep 09 21:46:25.798680 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.798674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:25.800394 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.800369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw" event={"ID":"33afc968-7590-4088-b21b-73b69e6008fc","Type":"ContainerStarted","Data":"a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43"}
Sep 09 21:46:25.800749 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.800421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw" event={"ID":"33afc968-7590-4088-b21b-73b69e6008fc","Type":"ContainerStarted","Data":"b8284c38ab7a1ab3634f767ae20bb0be403c1cc055d1de5a5f7454d217e9b9fc"}
Sep 09 21:46:25.800749 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.800487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:25.801678 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.801652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t" event={"ID":"9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba","Type":"ContainerStarted","Data":"26a58a702f747a7c97f4b7d5bbf9cbfa0355b86ab3cd985fe974b66f15eb190b"}
Sep 09 21:46:25.818368 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.818310    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl" podStartSLOduration=1.81828761 podStartE2EDuration="1.81828761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:25.814981124 +0000 UTC m=+791.300864164" watchObservedRunningTime="2026-09-09 21:46:25.81828761 +0000 UTC m=+791.304170649"
Sep 09 21:46:25.833828 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:25.833765    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw" podStartSLOduration=1.833746793 podStartE2EDuration="1.833746793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:25.830545741 +0000 UTC m=+791.316428781" watchObservedRunningTime="2026-09-09 21:46:25.833746793 +0000 UTC m=+791.319629833"
Sep 09 21:46:28.820967 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:28.820878    2563 generic.go:356] "Generic (PLEG): container finished" podID="9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba" containerID="9f1828a7511ba4729061fdf9ea57c8648289504fb8d471306955739142b250d9" exitCode=0
Sep 09 21:46:28.821333 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:28.820961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t" event={"ID":"9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba","Type":"ContainerDied","Data":"9f1828a7511ba4729061fdf9ea57c8648289504fb8d471306955739142b250d9"}
Sep 09 21:46:29.967353 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:29.967328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"
Sep 09 21:46:30.082946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:30.082843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba-maas-api-service-ca\") pod \"9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba\" (UID: \"9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba\") "
Sep 09 21:46:30.083219 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:30.083193    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba" (UID: "9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:30.183777 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:30.183707    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:46:30.832543 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:30.832516    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"
Sep 09 21:46:30.832756 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:30.832516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t" event={"ID":"9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba","Type":"ContainerDied","Data":"26a58a702f747a7c97f4b7d5bbf9cbfa0355b86ab3cd985fe974b66f15eb190b"}
Sep 09 21:46:30.832756 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:30.832654    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a58a702f747a7c97f4b7d5bbf9cbfa0355b86ab3cd985fe974b66f15eb190b"
Sep 09 21:46:36.811314 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:36.811228    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:46:36.811811 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:36.811316    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:46:41.591861 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.591827    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"]
Sep 09 21:46:41.592266 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.592047    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm" podUID="6332fd8b-3630-4baa-bad2-b647bbc517de" containerName="payload-processing" containerID="cri-o://7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb" gracePeriod=30
Sep 09 21:46:41.600598 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.600562    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"]
Sep 09 21:46:41.600908 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.600850    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq" podUID="d8437a2a-16ac-4c0d-8632-bd08177fd537" containerName="payload-pre-processing" containerID="cri-o://a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2" gracePeriod=30
Sep 09 21:46:41.758349 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.758286    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"]
Sep 09 21:46:41.761029 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.760979    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04d63732-lz88t"]
Sep 09 21:46:41.853621 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.853575    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:41.892259 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.892230    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:41.893314 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.893286    2563 generic.go:356] "Generic (PLEG): container finished" podID="d8437a2a-16ac-4c0d-8632-bd08177fd537" containerID="a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2" exitCode=0
Sep 09 21:46:41.893615 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.893367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq" event={"ID":"d8437a2a-16ac-4c0d-8632-bd08177fd537","Type":"ContainerDied","Data":"a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2"}
Sep 09 21:46:41.893615 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.893433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq" event={"ID":"d8437a2a-16ac-4c0d-8632-bd08177fd537","Type":"ContainerDied","Data":"a7f81e19bcd4abc58bb71b5ac584c1cd7a4341aa54cbd7692a99668ba0fb27c0"}
Sep 09 21:46:41.893615 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.893469    2563 scope.go:117] "RemoveContainer" containerID="a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2"
Sep 09 21:46:41.894888 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.894801    2563 generic.go:356] "Generic (PLEG): container finished" podID="6332fd8b-3630-4baa-bad2-b647bbc517de" containerID="7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb" exitCode=0
Sep 09 21:46:41.894888 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.894854    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"
Sep 09 21:46:41.895020 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.894856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm" event={"ID":"6332fd8b-3630-4baa-bad2-b647bbc517de","Type":"ContainerDied","Data":"7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb"}
Sep 09 21:46:41.895020 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.894955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm" event={"ID":"6332fd8b-3630-4baa-bad2-b647bbc517de","Type":"ContainerDied","Data":"d1c965fb9630ab7900e93337500743f378efabf0554587d7ab52b1b0126df544"}
Sep 09 21:46:41.896907 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.896886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6332fd8b-3630-4baa-bad2-b647bbc517de-plugins-config-volume\") pod \"6332fd8b-3630-4baa-bad2-b647bbc517de\" (UID: \"6332fd8b-3630-4baa-bad2-b647bbc517de\") "
Sep 09 21:46:41.897010 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.896947    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb6mc\" (UniqueName: \"kubernetes.io/projected/6332fd8b-3630-4baa-bad2-b647bbc517de-kube-api-access-tb6mc\") pod \"6332fd8b-3630-4baa-bad2-b647bbc517de\" (UID: \"6332fd8b-3630-4baa-bad2-b647bbc517de\") "
Sep 09 21:46:41.897234 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.897212    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6332fd8b-3630-4baa-bad2-b647bbc517de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6332fd8b-3630-4baa-bad2-b647bbc517de" (UID: "6332fd8b-3630-4baa-bad2-b647bbc517de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:41.897364 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.897334    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6332fd8b-3630-4baa-bad2-b647bbc517de-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:46:41.899772 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.899710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6332fd8b-3630-4baa-bad2-b647bbc517de-kube-api-access-tb6mc" (OuterVolumeSpecName: "kube-api-access-tb6mc") pod "6332fd8b-3630-4baa-bad2-b647bbc517de" (UID: "6332fd8b-3630-4baa-bad2-b647bbc517de"). InnerVolumeSpecName "kube-api-access-tb6mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:46:41.905329 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.905307    2563 scope.go:117] "RemoveContainer" containerID="a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2"
Sep 09 21:46:41.905639 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:46:41.905620    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2\": container with ID starting with a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2 not found: ID does not exist" containerID="a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2"
Sep 09 21:46:41.905703 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.905649    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2"} err="failed to get container status \"a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2\": rpc error: code = NotFound desc = could not find container \"a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2\": container with ID starting with a677797e4f74c325695e8860f4218f02f3ec54e423f4f4bb05d26448cf165da2 not found: ID does not exist"
Sep 09 21:46:41.905703 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.905665    2563 scope.go:117] "RemoveContainer" containerID="7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb"
Sep 09 21:46:41.917748 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.917730    2563 scope.go:117] "RemoveContainer" containerID="7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb"
Sep 09 21:46:41.918087 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:46:41.918064    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb\": container with ID starting with 7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb not found: ID does not exist" containerID="7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb"
Sep 09 21:46:41.918205 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.918091    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb"} err="failed to get container status \"7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb\": rpc error: code = NotFound desc = could not find container \"7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb\": container with ID starting with 7f938f8d666ec454a3c220a0be86569b391feae3176e7813d1e96be29f4c3bcb not found: ID does not exist"
Sep 09 21:46:41.998676 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.998634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8437a2a-16ac-4c0d-8632-bd08177fd537-plugins-config-volume\") pod \"d8437a2a-16ac-4c0d-8632-bd08177fd537\" (UID: \"d8437a2a-16ac-4c0d-8632-bd08177fd537\") "
Sep 09 21:46:41.998676 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.998677    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbhr4\" (UniqueName: \"kubernetes.io/projected/d8437a2a-16ac-4c0d-8632-bd08177fd537-kube-api-access-kbhr4\") pod \"d8437a2a-16ac-4c0d-8632-bd08177fd537\" (UID: \"d8437a2a-16ac-4c0d-8632-bd08177fd537\") "
Sep 09 21:46:41.998964 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.998906    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6mc\" (UniqueName: \"kubernetes.io/projected/6332fd8b-3630-4baa-bad2-b647bbc517de-kube-api-access-tb6mc\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:46:41.999064 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:41.999040    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8437a2a-16ac-4c0d-8632-bd08177fd537-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8437a2a-16ac-4c0d-8632-bd08177fd537" (UID: "d8437a2a-16ac-4c0d-8632-bd08177fd537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:42.000961 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:42.000937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8437a2a-16ac-4c0d-8632-bd08177fd537-kube-api-access-kbhr4" (OuterVolumeSpecName: "kube-api-access-kbhr4") pod "d8437a2a-16ac-4c0d-8632-bd08177fd537" (UID: "d8437a2a-16ac-4c0d-8632-bd08177fd537"). InnerVolumeSpecName "kube-api-access-kbhr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:46:42.099531 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:42.099486    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8437a2a-16ac-4c0d-8632-bd08177fd537-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:46:42.099531 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:42.099523    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhr4\" (UniqueName: \"kubernetes.io/projected/d8437a2a-16ac-4c0d-8632-bd08177fd537-kube-api-access-kbhr4\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:46:42.220639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:42.220599    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"]
Sep 09 21:46:42.224244 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:42.224212    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04d63732-5d467bccd9-pxkkm"]
Sep 09 21:46:42.900966 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:42.900886    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"
Sep 09 21:46:42.928172 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:42.928140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"]
Sep 09 21:46:42.937869 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:42.937837    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04d63732-8865dc7c8-c6kvq"]
Sep 09 21:46:43.046977 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:43.046941    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6332fd8b-3630-4baa-bad2-b647bbc517de" path="/var/lib/kubelet/pods/6332fd8b-3630-4baa-bad2-b647bbc517de/volumes"
Sep 09 21:46:43.047293 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:43.047280    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba" path="/var/lib/kubelet/pods/9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba/volumes"
Sep 09 21:46:43.047616 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:43.047604    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8437a2a-16ac-4c0d-8632-bd08177fd537" path="/var/lib/kubelet/pods/d8437a2a-16ac-4c0d-8632-bd08177fd537/volumes"
Sep 09 21:46:47.068029 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.067997    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"]
Sep 09 21:46:47.068396 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068386    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6332fd8b-3630-4baa-bad2-b647bbc517de" containerName="payload-processing"
Sep 09 21:46:47.068438 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068397    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6332fd8b-3630-4baa-bad2-b647bbc517de" containerName="payload-processing"
Sep 09 21:46:47.068438 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068406    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8437a2a-16ac-4c0d-8632-bd08177fd537" containerName="payload-pre-processing"
Sep 09 21:46:47.068438 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068411    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8437a2a-16ac-4c0d-8632-bd08177fd537" containerName="payload-pre-processing"
Sep 09 21:46:47.068438 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068418    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba" containerName="revoke-keys"
Sep 09 21:46:47.068438 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068423    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba" containerName="revoke-keys"
Sep 09 21:46:47.068584 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068495    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6332fd8b-3630-4baa-bad2-b647bbc517de" containerName="payload-processing"
Sep 09 21:46:47.068584 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068504    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bcfbdc9-ceee-45b9-8a8c-6078b10c8fba" containerName="revoke-keys"
Sep 09 21:46:47.068584 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.068511    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8437a2a-16ac-4c0d-8632-bd08177fd537" containerName="payload-pre-processing"
Sep 09 21:46:47.073115 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.073099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.076056 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.076035    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-98c7fb47-dockercfg-cptq9\""
Sep 09 21:46:47.076056 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.076050    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-98c7fb47\""
Sep 09 21:46:47.084624 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.084604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"]
Sep 09 21:46:47.096802 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.096781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"]
Sep 09 21:46:47.100454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.100433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.110473 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.110452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"]
Sep 09 21:46:47.151475 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.151440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdf6d\" (UniqueName: \"kubernetes.io/projected/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-kube-api-access-gdf6d\") pod \"payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9\" (UID: \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\") " pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.151475 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.151479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13d12a3-da0b-4228-a836-6bb76541c06f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv\" (UID: \"e13d12a3-da0b-4228-a836-6bb76541c06f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.151764 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.151585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq75c\" (UniqueName: \"kubernetes.io/projected/e13d12a3-da0b-4228-a836-6bb76541c06f-kube-api-access-bq75c\") pod \"payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv\" (UID: \"e13d12a3-da0b-4228-a836-6bb76541c06f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.151764 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.151647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-plugins-config-volume\") pod \"payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9\" (UID: \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\") " pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.252646 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.252587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdf6d\" (UniqueName: \"kubernetes.io/projected/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-kube-api-access-gdf6d\") pod \"payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9\" (UID: \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\") " pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.252894 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.252655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13d12a3-da0b-4228-a836-6bb76541c06f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv\" (UID: \"e13d12a3-da0b-4228-a836-6bb76541c06f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.252894 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.252770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq75c\" (UniqueName: \"kubernetes.io/projected/e13d12a3-da0b-4228-a836-6bb76541c06f-kube-api-access-bq75c\") pod \"payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv\" (UID: \"e13d12a3-da0b-4228-a836-6bb76541c06f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.252894 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.252814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-plugins-config-volume\") pod \"payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9\" (UID: \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\") " pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.253382 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.253362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13d12a3-da0b-4228-a836-6bb76541c06f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv\" (UID: \"e13d12a3-da0b-4228-a836-6bb76541c06f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.253483 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.253466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-plugins-config-volume\") pod \"payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9\" (UID: \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\") " pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.264359 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.264328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdf6d\" (UniqueName: \"kubernetes.io/projected/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-kube-api-access-gdf6d\") pod \"payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9\" (UID: \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\") " pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.264525 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.264444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq75c\" (UniqueName: \"kubernetes.io/projected/e13d12a3-da0b-4228-a836-6bb76541c06f-kube-api-access-bq75c\") pod \"payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv\" (UID: \"e13d12a3-da0b-4228-a836-6bb76541c06f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.384592 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.384553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.411927 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.411898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.660437 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.660410    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"]
Sep 09 21:46:47.662460 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:46:47.662434    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13d12a3_da0b_4228_a836_6bb76541c06f.slice/crio-2c60a04f14fe535c0a582ca7bb913c3a277a31195bcc9273af30c2f17fe95b34 WatchSource:0}: Error finding container 2c60a04f14fe535c0a582ca7bb913c3a277a31195bcc9273af30c2f17fe95b34: Status 404 returned error can't find the container with id 2c60a04f14fe535c0a582ca7bb913c3a277a31195bcc9273af30c2f17fe95b34
Sep 09 21:46:47.696001 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.695969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"]
Sep 09 21:46:47.698862 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:46:47.698837    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5fb67c_d1a2_45a6_836f_e2e3dd16e632.slice/crio-16d7379848478a278214293144d76f01a0f3f1c72c7022cb318b7c7a7dec8938 WatchSource:0}: Error finding container 16d7379848478a278214293144d76f01a0f3f1c72c7022cb318b7c7a7dec8938: Status 404 returned error can't find the container with id 16d7379848478a278214293144d76f01a0f3f1c72c7022cb318b7c7a7dec8938
Sep 09 21:46:47.930588 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.930488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9" event={"ID":"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632","Type":"ContainerStarted","Data":"86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511"}
Sep 09 21:46:47.930588 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.930560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:47.930588 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.930584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9" event={"ID":"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632","Type":"ContainerStarted","Data":"16d7379848478a278214293144d76f01a0f3f1c72c7022cb318b7c7a7dec8938"}
Sep 09 21:46:47.932194 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.932170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv" event={"ID":"e13d12a3-da0b-4228-a836-6bb76541c06f","Type":"ContainerStarted","Data":"43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52"}
Sep 09 21:46:47.932302 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.932218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv" event={"ID":"e13d12a3-da0b-4228-a836-6bb76541c06f","Type":"ContainerStarted","Data":"2c60a04f14fe535c0a582ca7bb913c3a277a31195bcc9273af30c2f17fe95b34"}
Sep 09 21:46:47.932302 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.932277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:46:47.950355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.950306    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9" podStartSLOduration=0.950290548 podStartE2EDuration="950.290548ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:47.949370915 +0000 UTC m=+813.435253955" watchObservedRunningTime="2026-09-09 21:46:47.950290548 +0000 UTC m=+813.436173585"
Sep 09 21:46:47.969070 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:47.968886    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv" podStartSLOduration=0.968868693 podStartE2EDuration="968.868693ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:47.968490276 +0000 UTC m=+813.454373339" watchObservedRunningTime="2026-09-09 21:46:47.968868693 +0000 UTC m=+813.454751740"
Sep 09 21:46:58.940682 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:58.940642    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:46:58.941104 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:46:58.941009    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:47:05.228929 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.228895    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"]
Sep 09 21:47:05.236627 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.236607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.246897 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.246878    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"kube-root-ca.crt\""
Sep 09 21:47:05.248279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.248262    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"openshift-service-ca.crt\""
Sep 09 21:47:05.248411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.248395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"default-dockercfg-h6wml\""
Sep 09 21:47:05.248457 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.248444    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"simulator-w4-3fa986-kserve-self-signed-certs\""
Sep 09 21:47:05.258529 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.258503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"]
Sep 09 21:47:05.330168 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.330128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-home\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.330168 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.330170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-dshm\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.330435 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.330268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-model-cache\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.330435 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.330328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tmp-dir\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.330435 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.330385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgq5c\" (UniqueName: \"kubernetes.io/projected/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-kube-api-access-bgq5c\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.330435 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.330431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tls-certs\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.431935 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.431900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-model-cache\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.432155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.431956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tmp-dir\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.432155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.431990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq5c\" (UniqueName: \"kubernetes.io/projected/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-kube-api-access-bgq5c\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.432155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.432027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tls-certs\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.432155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.432088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-home\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.432155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.432119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-dshm\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.432436 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.432370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-model-cache\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.432480 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.432445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tmp-dir\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.432569 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.432548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-home\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.434483 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.434459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-dshm\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.434606 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.434577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tls-certs\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.455902 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.455871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgq5c\" (UniqueName: \"kubernetes.io/projected/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-kube-api-access-bgq5c\") pod \"simulator-w4-3fa986-kserve-66f859d6c8-87m8m\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") " pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.546131 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.546044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:05.829253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:05.829230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"]
Sep 09 21:47:05.831003 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:47:05.830976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf3cc34_b7c5_4bb8_80e6_c64394ca95c6.slice/crio-6d4c96e50e7a3848223e8a339e439b47cc3adbaed7663bf292a86bb3fe8f4825 WatchSource:0}: Error finding container 6d4c96e50e7a3848223e8a339e439b47cc3adbaed7663bf292a86bb3fe8f4825: Status 404 returned error can't find the container with id 6d4c96e50e7a3848223e8a339e439b47cc3adbaed7663bf292a86bb3fe8f4825
Sep 09 21:47:06.027712 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:06.027672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m" event={"ID":"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6","Type":"ContainerStarted","Data":"66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535"}
Sep 09 21:47:06.027712 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:06.027748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m" event={"ID":"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6","Type":"ContainerStarted","Data":"6d4c96e50e7a3848223e8a339e439b47cc3adbaed7663bf292a86bb3fe8f4825"}
Sep 09 21:47:06.050324 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:06.050266    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m" podStartSLOduration=1.050249897 podStartE2EDuration="1.050249897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:06.04957004 +0000 UTC m=+831.535453080" watchObservedRunningTime="2026-09-09 21:47:06.050249897 +0000 UTC m=+831.536132936"
Sep 09 21:47:10.776468 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:10.776424    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"]
Sep 09 21:47:10.780790 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:10.780767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"
Sep 09 21:47:10.788010 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:10.787982    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:47:10.797516 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:10.797486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"]
Sep 09 21:47:10.881405 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:10.881371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e9855b-7be9-4ce2-acc4-f6f816b31a79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf\" (UID: \"87e9855b-7be9-4ce2-acc4-f6f816b31a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"
Sep 09 21:47:10.982125 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:10.982078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e9855b-7be9-4ce2-acc4-f6f816b31a79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf\" (UID: \"87e9855b-7be9-4ce2-acc4-f6f816b31a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"
Sep 09 21:47:10.982699 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:10.982676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e9855b-7be9-4ce2-acc4-f6f816b31a79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf\" (UID: \"87e9855b-7be9-4ce2-acc4-f6f816b31a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"
Sep 09 21:47:11.091945 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:11.091859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"
Sep 09 21:47:11.392703 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:11.392665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"]
Sep 09 21:47:11.397124 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:47:11.397097    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e9855b_7be9_4ce2_acc4_f6f816b31a79.slice/crio-6dd5ea96f5646b734b3aaea3cb154d04b0163a1e5b513cf704a3baca89b0123a WatchSource:0}: Error finding container 6dd5ea96f5646b734b3aaea3cb154d04b0163a1e5b513cf704a3baca89b0123a: Status 404 returned error can't find the container with id 6dd5ea96f5646b734b3aaea3cb154d04b0163a1e5b513cf704a3baca89b0123a
Sep 09 21:47:12.061613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:12.061506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf" event={"ID":"87e9855b-7be9-4ce2-acc4-f6f816b31a79","Type":"ContainerStarted","Data":"76e1d60e25a4e6a5c9d4d6604e6d70de8f1817a8f1b1caef6d6ce39477d26cf8"}
Sep 09 21:47:12.062108 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:12.061614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf" event={"ID":"87e9855b-7be9-4ce2-acc4-f6f816b31a79","Type":"ContainerStarted","Data":"6dd5ea96f5646b734b3aaea3cb154d04b0163a1e5b513cf704a3baca89b0123a"}
Sep 09 21:47:12.080129 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:12.080068    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf" podStartSLOduration=2.080052354 podStartE2EDuration="2.080052354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:12.079669845 +0000 UTC m=+837.565552884" watchObservedRunningTime="2026-09-09 21:47:12.080052354 +0000 UTC m=+837.565935391"
Sep 09 21:47:13.068705 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:13.068668    2563 generic.go:356] "Generic (PLEG): container finished" podID="87e9855b-7be9-4ce2-acc4-f6f816b31a79" containerID="76e1d60e25a4e6a5c9d4d6604e6d70de8f1817a8f1b1caef6d6ce39477d26cf8" exitCode=0
Sep 09 21:47:13.069107 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:13.068753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf" event={"ID":"87e9855b-7be9-4ce2-acc4-f6f816b31a79","Type":"ContainerDied","Data":"76e1d60e25a4e6a5c9d4d6604e6d70de8f1817a8f1b1caef6d6ce39477d26cf8"}
Sep 09 21:47:14.208671 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:14.208644    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"
Sep 09 21:47:14.211709 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:14.211692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e9855b-7be9-4ce2-acc4-f6f816b31a79-maas-api-service-ca\") pod \"87e9855b-7be9-4ce2-acc4-f6f816b31a79\" (UID: \"87e9855b-7be9-4ce2-acc4-f6f816b31a79\") "
Sep 09 21:47:14.211984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:14.211964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e9855b-7be9-4ce2-acc4-f6f816b31a79-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87e9855b-7be9-4ce2-acc4-f6f816b31a79" (UID: "87e9855b-7be9-4ce2-acc4-f6f816b31a79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:47:14.312829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:14.312797    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e9855b-7be9-4ce2-acc4-f6f816b31a79-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:15.080536 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:15.080502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf" event={"ID":"87e9855b-7be9-4ce2-acc4-f6f816b31a79","Type":"ContainerDied","Data":"6dd5ea96f5646b734b3aaea3cb154d04b0163a1e5b513cf704a3baca89b0123a"}
Sep 09 21:47:15.080661 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:15.080556    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd5ea96f5646b734b3aaea3cb154d04b0163a1e5b513cf704a3baca89b0123a"
Sep 09 21:47:15.080661 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:15.080515    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"
Sep 09 21:47:15.546604 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:15.546573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:15.547081 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:15.546884    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:15.559244 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:15.559222    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:16.096702 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:16.096671    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 21:47:17.540880 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.540838    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"]
Sep 09 21:47:17.541305 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.541118    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9" podUID="8a5fb67c-d1a2-45a6-836f-e2e3dd16e632" containerName="payload-processing" containerID="cri-o://86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511" gracePeriod=30
Sep 09 21:47:17.553123 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.553090    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"]
Sep 09 21:47:17.553397 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.553345    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv" podUID="e13d12a3-da0b-4228-a836-6bb76541c06f" containerName="payload-pre-processing" containerID="cri-o://43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52" gracePeriod=30
Sep 09 21:47:17.788246 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.788184    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"]
Sep 09 21:47:17.791463 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.791383    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98c7fb47-s75zf"]
Sep 09 21:47:17.843426 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.843401    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:47:17.846467 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.846447    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:47:17.944843 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.944813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-plugins-config-volume\") pod \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\" (UID: \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\") "
Sep 09 21:47:17.945027 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.944912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13d12a3-da0b-4228-a836-6bb76541c06f-plugins-config-volume\") pod \"e13d12a3-da0b-4228-a836-6bb76541c06f\" (UID: \"e13d12a3-da0b-4228-a836-6bb76541c06f\") "
Sep 09 21:47:17.945027 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.945014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdf6d\" (UniqueName: \"kubernetes.io/projected/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-kube-api-access-gdf6d\") pod \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\" (UID: \"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632\") "
Sep 09 21:47:17.945101 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.945047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq75c\" (UniqueName: \"kubernetes.io/projected/e13d12a3-da0b-4228-a836-6bb76541c06f-kube-api-access-bq75c\") pod \"e13d12a3-da0b-4228-a836-6bb76541c06f\" (UID: \"e13d12a3-da0b-4228-a836-6bb76541c06f\") "
Sep 09 21:47:17.945265 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.945244    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a5fb67c-d1a2-45a6-836f-e2e3dd16e632" (UID: "8a5fb67c-d1a2-45a6-836f-e2e3dd16e632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:47:17.945303 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.945256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13d12a3-da0b-4228-a836-6bb76541c06f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e13d12a3-da0b-4228-a836-6bb76541c06f" (UID: "e13d12a3-da0b-4228-a836-6bb76541c06f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:47:17.945362 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.945349    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:17.945398 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.945367    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13d12a3-da0b-4228-a836-6bb76541c06f-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:17.947354 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.947333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-kube-api-access-gdf6d" (OuterVolumeSpecName: "kube-api-access-gdf6d") pod "8a5fb67c-d1a2-45a6-836f-e2e3dd16e632" (UID: "8a5fb67c-d1a2-45a6-836f-e2e3dd16e632"). InnerVolumeSpecName "kube-api-access-gdf6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:17.947423 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:17.947363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13d12a3-da0b-4228-a836-6bb76541c06f-kube-api-access-bq75c" (OuterVolumeSpecName: "kube-api-access-bq75c") pod "e13d12a3-da0b-4228-a836-6bb76541c06f" (UID: "e13d12a3-da0b-4228-a836-6bb76541c06f"). InnerVolumeSpecName "kube-api-access-bq75c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:18.046186 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.046097    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdf6d\" (UniqueName: \"kubernetes.io/projected/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632-kube-api-access-gdf6d\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:18.046186 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.046127    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq75c\" (UniqueName: \"kubernetes.io/projected/e13d12a3-da0b-4228-a836-6bb76541c06f-kube-api-access-bq75c\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:18.097279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.097238    2563 generic.go:356] "Generic (PLEG): container finished" podID="e13d12a3-da0b-4228-a836-6bb76541c06f" containerID="43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52" exitCode=0
Sep 09 21:47:18.097279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.097280    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"
Sep 09 21:47:18.097515 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.097326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv" event={"ID":"e13d12a3-da0b-4228-a836-6bb76541c06f","Type":"ContainerDied","Data":"43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52"}
Sep 09 21:47:18.097515 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.097379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv" event={"ID":"e13d12a3-da0b-4228-a836-6bb76541c06f","Type":"ContainerDied","Data":"2c60a04f14fe535c0a582ca7bb913c3a277a31195bcc9273af30c2f17fe95b34"}
Sep 09 21:47:18.097515 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.097410    2563 scope.go:117] "RemoveContainer" containerID="43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52"
Sep 09 21:47:18.098729 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.098686    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a5fb67c-d1a2-45a6-836f-e2e3dd16e632" containerID="86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511" exitCode=0
Sep 09 21:47:18.098863 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.098761    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"
Sep 09 21:47:18.098863 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.098787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9" event={"ID":"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632","Type":"ContainerDied","Data":"86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511"}
Sep 09 21:47:18.098863 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.098837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9" event={"ID":"8a5fb67c-d1a2-45a6-836f-e2e3dd16e632","Type":"ContainerDied","Data":"16d7379848478a278214293144d76f01a0f3f1c72c7022cb318b7c7a7dec8938"}
Sep 09 21:47:18.107650 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.107611    2563 scope.go:117] "RemoveContainer" containerID="43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52"
Sep 09 21:47:18.108002 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:47:18.107980    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52\": container with ID starting with 43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52 not found: ID does not exist" containerID="43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52"
Sep 09 21:47:18.108110 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.108013    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52"} err="failed to get container status \"43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52\": rpc error: code = NotFound desc = could not find container \"43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52\": container with ID starting with 43d8bdaf1bd6b58ccf4cbbfe4f8ed9a1c818b12cb937010d197fd04282f7eb52 not found: ID does not exist"
Sep 09 21:47:18.108110 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.108035    2563 scope.go:117] "RemoveContainer" containerID="86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511"
Sep 09 21:47:18.116961 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.116944    2563 scope.go:117] "RemoveContainer" containerID="86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511"
Sep 09 21:47:18.117281 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:47:18.117261    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511\": container with ID starting with 86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511 not found: ID does not exist" containerID="86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511"
Sep 09 21:47:18.117333 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.117290    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511"} err="failed to get container status \"86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511\": rpc error: code = NotFound desc = could not find container \"86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511\": container with ID starting with 86314ccb865ed73e11d20771f00245956c46fa6898b042e52e3496a641840511 not found: ID does not exist"
Sep 09 21:47:18.126892 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.126860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"]
Sep 09 21:47:18.130829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.130804    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-98c7fb47-d7cccc7db-mp8xv"]
Sep 09 21:47:18.147893 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.147861    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"]
Sep 09 21:47:18.158272 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:18.158243    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-98c7fb47-549cbfd9d6-rlwr9"]
Sep 09 21:47:19.046962 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:19.046930    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e9855b-7be9-4ce2-acc4-f6f816b31a79" path="/var/lib/kubelet/pods/87e9855b-7be9-4ce2-acc4-f6f816b31a79/volumes"
Sep 09 21:47:19.047360 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:19.047341    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a5fb67c-d1a2-45a6-836f-e2e3dd16e632" path="/var/lib/kubelet/pods/8a5fb67c-d1a2-45a6-836f-e2e3dd16e632/volumes"
Sep 09 21:47:19.047703 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:19.047691    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13d12a3-da0b-4228-a836-6bb76541c06f" path="/var/lib/kubelet/pods/e13d12a3-da0b-4228-a836-6bb76541c06f/volumes"
Sep 09 21:47:23.011298 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011260    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"]
Sep 09 21:47:23.011674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011664    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5fb67c-d1a2-45a6-836f-e2e3dd16e632" containerName="payload-processing"
Sep 09 21:47:23.011758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011675    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5fb67c-d1a2-45a6-836f-e2e3dd16e632" containerName="payload-processing"
Sep 09 21:47:23.011758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011683    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e9855b-7be9-4ce2-acc4-f6f816b31a79" containerName="revoke-keys"
Sep 09 21:47:23.011758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011688    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e9855b-7be9-4ce2-acc4-f6f816b31a79" containerName="revoke-keys"
Sep 09 21:47:23.011758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011706    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13d12a3-da0b-4228-a836-6bb76541c06f" containerName="payload-pre-processing"
Sep 09 21:47:23.011758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011713    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13d12a3-da0b-4228-a836-6bb76541c06f" containerName="payload-pre-processing"
Sep 09 21:47:23.011919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011817    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e9855b-7be9-4ce2-acc4-f6f816b31a79" containerName="revoke-keys"
Sep 09 21:47:23.011919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011827    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5fb67c-d1a2-45a6-836f-e2e3dd16e632" containerName="payload-processing"
Sep 09 21:47:23.011919 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.011835    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13d12a3-da0b-4228-a836-6bb76541c06f" containerName="payload-pre-processing"
Sep 09 21:47:23.016306 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.016289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:23.020079 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.020056    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b27eb4af\""
Sep 09 21:47:23.020079 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.020072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b27eb4af-dockercfg-rlssx\""
Sep 09 21:47:23.037704 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.037672    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"]
Sep 09 21:47:23.056495 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.056460    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"]
Sep 09 21:47:23.060855 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.060832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:23.076559 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.076530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"]
Sep 09 21:47:23.096249 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.096209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd\" (UID: \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\") " pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:23.096411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.096309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95\" (UID: \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:23.096411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.096347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stksh\" (UniqueName: \"kubernetes.io/projected/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-kube-api-access-stksh\") pod \"payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95\" (UID: \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:23.096411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.096364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dvf\" (UniqueName: \"kubernetes.io/projected/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-kube-api-access-96dvf\") pod \"payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd\" (UID: \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\") " pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:23.196920 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.196889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd\" (UID: \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\") " pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:23.197155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.196968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95\" (UID: \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:23.197155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.196999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stksh\" (UniqueName: \"kubernetes.io/projected/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-kube-api-access-stksh\") pod \"payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95\" (UID: \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:23.197155 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.197018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dvf\" (UniqueName: \"kubernetes.io/projected/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-kube-api-access-96dvf\") pod \"payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd\" (UID: \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\") " pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:23.197594 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.197571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd\" (UID: \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\") " pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:23.197761 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.197742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95\" (UID: \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:23.212930 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.212898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stksh\" (UniqueName: \"kubernetes.io/projected/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-kube-api-access-stksh\") pod \"payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95\" (UID: \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:23.213494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.213475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dvf\" (UniqueName: \"kubernetes.io/projected/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-kube-api-access-96dvf\") pod \"payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd\" (UID: \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\") " pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:23.326907 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.326822    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:23.373611 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.373552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:23.607922 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.607893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"]
Sep 09 21:47:23.609874 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:47:23.609848    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25ad8fe_b2d9_4fc6_8bbe_9f1d5cdda00e.slice/crio-1bfc1f01ae79d0ce457e1bb99373a6d9e5a7d592274f509d54a3e1f8f276cb3a WatchSource:0}: Error finding container 1bfc1f01ae79d0ce457e1bb99373a6d9e5a7d592274f509d54a3e1f8f276cb3a: Status 404 returned error can't find the container with id 1bfc1f01ae79d0ce457e1bb99373a6d9e5a7d592274f509d54a3e1f8f276cb3a
Sep 09 21:47:23.665196 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:23.665172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"]
Sep 09 21:47:23.667675 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:47:23.667650    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f62cfe_70cc_4a3c_8c8d_b6c617a2099d.slice/crio-9ec3514da855f25f1210a763d9ed9dca0aa38ca0cfc1148f73f59ab866edf679 WatchSource:0}: Error finding container 9ec3514da855f25f1210a763d9ed9dca0aa38ca0cfc1148f73f59ab866edf679: Status 404 returned error can't find the container with id 9ec3514da855f25f1210a763d9ed9dca0aa38ca0cfc1148f73f59ab866edf679
Sep 09 21:47:24.135785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:24.135398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95" event={"ID":"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e","Type":"ContainerStarted","Data":"bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4"}
Sep 09 21:47:24.135785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:24.135593    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:24.135785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:24.135628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95" event={"ID":"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e","Type":"ContainerStarted","Data":"1bfc1f01ae79d0ce457e1bb99373a6d9e5a7d592274f509d54a3e1f8f276cb3a"}
Sep 09 21:47:24.137133 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:24.137096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd" event={"ID":"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d","Type":"ContainerStarted","Data":"2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1"}
Sep 09 21:47:24.137255 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:24.137144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd" event={"ID":"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d","Type":"ContainerStarted","Data":"9ec3514da855f25f1210a763d9ed9dca0aa38ca0cfc1148f73f59ab866edf679"}
Sep 09 21:47:24.137255 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:24.137173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:24.154315 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:24.154247    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95" podStartSLOduration=2.154227854 podStartE2EDuration="2.154227854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:24.15388062 +0000 UTC m=+849.639763695" watchObservedRunningTime="2026-09-09 21:47:24.154227854 +0000 UTC m=+849.640110893"
Sep 09 21:47:24.180211 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:24.180146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd" podStartSLOduration=1.180127891 podStartE2EDuration="1.180127891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:24.177123848 +0000 UTC m=+849.663006885" watchObservedRunningTime="2026-09-09 21:47:24.180127891 +0000 UTC m=+849.666010930"
Sep 09 21:47:35.145156 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:35.145123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:35.145566 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:35.145194    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:43.088822 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.088772    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"]
Sep 09 21:47:43.092585 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.092568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.095420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.095394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-390abc\"/\"default-dockercfg-pszs8\""
Sep 09 21:47:43.095420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.095412    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-390abc\"/\"kube-root-ca.crt\""
Sep 09 21:47:43.095592 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.095400    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-390abc\"/\"openshift-service-ca.crt\""
Sep 09 21:47:43.096647 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.096629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-390abc\"/\"simulator-w0-390abc-kserve-self-signed-certs\""
Sep 09 21:47:43.102048 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.102024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"]
Sep 09 21:47:43.194050 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.194016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-tmp-dir\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.194233 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.194058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-model-cache\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.194233 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.194125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5lk\" (UniqueName: \"kubernetes.io/projected/04e53c54-d895-4d11-800b-c0017ca20e3d-kube-api-access-fp5lk\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.194233 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.194189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-dshm\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.194233 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.194212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e53c54-d895-4d11-800b-c0017ca20e3d-tls-certs\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.194403 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.194277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-home\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.295466 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.295433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-tmp-dir\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.295466 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.295469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-model-cache\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.295766 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.295487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5lk\" (UniqueName: \"kubernetes.io/projected/04e53c54-d895-4d11-800b-c0017ca20e3d-kube-api-access-fp5lk\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.295766 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.295522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-dshm\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.295766 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.295540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e53c54-d895-4d11-800b-c0017ca20e3d-tls-certs\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.295766 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.295574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-home\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.295971 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.295947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-model-cache\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.296040 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.295977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-tmp-dir\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.296040 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.296020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-home\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.297975 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.297954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-dshm\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.298208 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.298192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e53c54-d895-4d11-800b-c0017ca20e3d-tls-certs\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.305264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.305237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5lk\" (UniqueName: \"kubernetes.io/projected/04e53c54-d895-4d11-800b-c0017ca20e3d-kube-api-access-fp5lk\") pod \"simulator-w0-390abc-kserve-685b9f47bf-6czs4\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") " pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.403943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.403892    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:43.678100 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:43.678066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"]
Sep 09 21:47:43.679695 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:47:43.679667    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04e53c54_d895_4d11_800b_c0017ca20e3d.slice/crio-f4f801db8499fc8447860c1e2616fa66f95814a893a16e162a269ba2c21c2d0a WatchSource:0}: Error finding container f4f801db8499fc8447860c1e2616fa66f95814a893a16e162a269ba2c21c2d0a: Status 404 returned error can't find the container with id f4f801db8499fc8447860c1e2616fa66f95814a893a16e162a269ba2c21c2d0a
Sep 09 21:47:44.243462 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:44.243426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4" event={"ID":"04e53c54-d895-4d11-800b-c0017ca20e3d","Type":"ContainerStarted","Data":"d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6"}
Sep 09 21:47:44.243462 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:44.243467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4" event={"ID":"04e53c54-d895-4d11-800b-c0017ca20e3d","Type":"ContainerStarted","Data":"f4f801db8499fc8447860c1e2616fa66f95814a893a16e162a269ba2c21c2d0a"}
Sep 09 21:47:44.263789 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:44.263712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4" podStartSLOduration=1.26369632 podStartE2EDuration="1.26369632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:44.260566276 +0000 UTC m=+869.746449330" watchObservedRunningTime="2026-09-09 21:47:44.26369632 +0000 UTC m=+869.749579357"
Sep 09 21:47:46.674164 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:46.674130    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"]
Sep 09 21:47:46.677871 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:46.677854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"
Sep 09 21:47:46.681516 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:46.681492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:47:46.688556 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:46.688533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"]
Sep 09 21:47:46.832743 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:46.832685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43cea44-3cec-4881-a842-3765f75f7030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk\" (UID: \"d43cea44-3cec-4881-a842-3765f75f7030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"
Sep 09 21:47:46.934514 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:46.934424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43cea44-3cec-4881-a842-3765f75f7030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk\" (UID: \"d43cea44-3cec-4881-a842-3765f75f7030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"
Sep 09 21:47:46.935271 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:46.935248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43cea44-3cec-4881-a842-3765f75f7030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk\" (UID: \"d43cea44-3cec-4881-a842-3765f75f7030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"
Sep 09 21:47:46.989816 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:46.989784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"
Sep 09 21:47:47.267016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:47.266992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"]
Sep 09 21:47:47.269144 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:47:47.269114    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43cea44_3cec_4881_a842_3765f75f7030.slice/crio-0a2ab247df23bf2945a0dcf98c2607747a38f814c97f0655241208842dcad357 WatchSource:0}: Error finding container 0a2ab247df23bf2945a0dcf98c2607747a38f814c97f0655241208842dcad357: Status 404 returned error can't find the container with id 0a2ab247df23bf2945a0dcf98c2607747a38f814c97f0655241208842dcad357
Sep 09 21:47:48.268244 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:48.268152    2563 generic.go:356] "Generic (PLEG): container finished" podID="d43cea44-3cec-4881-a842-3765f75f7030" containerID="5a1205b80d4192d52d63bf9a1cdd0507d3da48e06761c6bb01a26de635e0067a" exitCode=0
Sep 09 21:47:48.268622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:48.268239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk" event={"ID":"d43cea44-3cec-4881-a842-3765f75f7030","Type":"ContainerDied","Data":"5a1205b80d4192d52d63bf9a1cdd0507d3da48e06761c6bb01a26de635e0067a"}
Sep 09 21:47:48.268622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:48.268290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk" event={"ID":"d43cea44-3cec-4881-a842-3765f75f7030","Type":"ContainerStarted","Data":"0a2ab247df23bf2945a0dcf98c2607747a38f814c97f0655241208842dcad357"}
Sep 09 21:47:49.406130 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:49.406105    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"
Sep 09 21:47:49.562134 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:49.562034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43cea44-3cec-4881-a842-3765f75f7030-maas-api-service-ca\") pod \"d43cea44-3cec-4881-a842-3765f75f7030\" (UID: \"d43cea44-3cec-4881-a842-3765f75f7030\") "
Sep 09 21:47:49.562403 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:49.562380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43cea44-3cec-4881-a842-3765f75f7030-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d43cea44-3cec-4881-a842-3765f75f7030" (UID: "d43cea44-3cec-4881-a842-3765f75f7030"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:47:49.662957 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:49.662918    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43cea44-3cec-4881-a842-3765f75f7030-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:50.279796 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:50.279769    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"
Sep 09 21:47:50.279975 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:50.279767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk" event={"ID":"d43cea44-3cec-4881-a842-3765f75f7030","Type":"ContainerDied","Data":"0a2ab247df23bf2945a0dcf98c2607747a38f814c97f0655241208842dcad357"}
Sep 09 21:47:50.279975 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:50.279890    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a2ab247df23bf2945a0dcf98c2607747a38f814c97f0655241208842dcad357"
Sep 09 21:47:52.150740 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.150313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"]
Sep 09 21:47:52.151223 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.151010    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43cea44-3cec-4881-a842-3765f75f7030" containerName="revoke-keys"
Sep 09 21:47:52.151223 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.151030    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43cea44-3cec-4881-a842-3765f75f7030" containerName="revoke-keys"
Sep 09 21:47:52.151223 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.151126    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43cea44-3cec-4881-a842-3765f75f7030" containerName="revoke-keys"
Sep 09 21:47:52.159020 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.158869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.165993 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.165191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-66cedd\"/\"default-dockercfg-bl7xk\""
Sep 09 21:47:52.165993 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.165419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-66cedd\"/\"kube-root-ca.crt\""
Sep 09 21:47:52.165993 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.165586    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-66cedd\"/\"openshift-service-ca.crt\""
Sep 09 21:47:52.165993 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.165795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-66cedd\"/\"autores-model-66cedd-kserve-self-signed-certs\""
Sep 09 21:47:52.168095 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.167962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"]
Sep 09 21:47:52.187004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.186971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-dshm\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.187163 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.187042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-home\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.187163 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.187111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-model-cache\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.187163 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.187139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tls-certs\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.187310 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.187176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tmp-dir\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.187310 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.187217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snl8s\" (UniqueName: \"kubernetes.io/projected/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-kube-api-access-snl8s\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.287783 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.287730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-dshm\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.287984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.287811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-home\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.287984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.287866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-model-cache\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.287984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.287889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tls-certs\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.287984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.287911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tmp-dir\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.287984 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.287955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snl8s\" (UniqueName: \"kubernetes.io/projected/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-kube-api-access-snl8s\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.288310 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.288276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-model-cache\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.288412 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.288373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-home\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.288555 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.288524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tmp-dir\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.290511 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.290475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-dshm\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.290783 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.290766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tls-certs\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.313504 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.313477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snl8s\" (UniqueName: \"kubernetes.io/projected/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-kube-api-access-snl8s\") pod \"autores-model-66cedd-kserve-647df78cff-lpjth\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") " pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.475891 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.475793    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:47:52.785797 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:52.785768    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"]
Sep 09 21:47:52.786690 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:47:52.786666    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab1a079a_5eb4_46f8_8e5c_46f64978dd32.slice/crio-46d024068a41f5a7e72f6420668733b5e45a335e3badcb5b7ee0c6338559ad6b WatchSource:0}: Error finding container 46d024068a41f5a7e72f6420668733b5e45a335e3badcb5b7ee0c6338559ad6b: Status 404 returned error can't find the container with id 46d024068a41f5a7e72f6420668733b5e45a335e3badcb5b7ee0c6338559ad6b
Sep 09 21:47:53.299555 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.299523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth" event={"ID":"ab1a079a-5eb4-46f8-8e5c-46f64978dd32","Type":"ContainerStarted","Data":"505ce3184e160bbf3770a1a0085c1f86a5541cc0df680a1ea9f26394f9a7e4b1"}
Sep 09 21:47:53.299957 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.299568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth" event={"ID":"ab1a079a-5eb4-46f8-8e5c-46f64978dd32","Type":"ContainerStarted","Data":"46d024068a41f5a7e72f6420668733b5e45a335e3badcb5b7ee0c6338559ad6b"}
Sep 09 21:47:53.332993 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.332933    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth" podStartSLOduration=1.3329174240000001 podStartE2EDuration="1.332917424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:53.330709695 +0000 UTC m=+878.816592735" watchObservedRunningTime="2026-09-09 21:47:53.332917424 +0000 UTC m=+878.818800462"
Sep 09 21:47:53.404781 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.404745    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:53.404966 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.404803    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:53.417018 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.416992    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:53.634989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.634948    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"]
Sep 09 21:47:53.635354 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.635166    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd" podUID="31f62cfe-70cc-4a3c-8c8d-b6c617a2099d" containerName="payload-processing" containerID="cri-o://2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1" gracePeriod=30
Sep 09 21:47:53.683782 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.683732    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"]
Sep 09 21:47:53.684013 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.683989    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95" podUID="c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e" containerName="payload-pre-processing" containerID="cri-o://bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4" gracePeriod=30
Sep 09 21:47:53.961621 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.961598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:53.964354 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:53.964335    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:54.005253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.005218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96dvf\" (UniqueName: \"kubernetes.io/projected/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-kube-api-access-96dvf\") pod \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\" (UID: \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\") "
Sep 09 21:47:54.005419 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.005272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stksh\" (UniqueName: \"kubernetes.io/projected/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-kube-api-access-stksh\") pod \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\" (UID: \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\") "
Sep 09 21:47:54.005419 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.005310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-plugins-config-volume\") pod \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\" (UID: \"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e\") "
Sep 09 21:47:54.005419 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.005357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-plugins-config-volume\") pod \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\" (UID: \"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d\") "
Sep 09 21:47:54.006113 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.006009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e" (UID: "c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:47:54.006113 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.006018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31f62cfe-70cc-4a3c-8c8d-b6c617a2099d" (UID: "31f62cfe-70cc-4a3c-8c8d-b6c617a2099d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:47:54.007619 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.007594    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-kube-api-access-96dvf" (OuterVolumeSpecName: "kube-api-access-96dvf") pod "31f62cfe-70cc-4a3c-8c8d-b6c617a2099d" (UID: "31f62cfe-70cc-4a3c-8c8d-b6c617a2099d"). InnerVolumeSpecName "kube-api-access-96dvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:54.007751 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.007691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-kube-api-access-stksh" (OuterVolumeSpecName: "kube-api-access-stksh") pod "c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e" (UID: "c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e"). InnerVolumeSpecName "kube-api-access-stksh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:54.034204 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.034170    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"]
Sep 09 21:47:54.039577 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.039548    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b27eb4af-8snmk"]
Sep 09 21:47:54.107194 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.107156    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96dvf\" (UniqueName: \"kubernetes.io/projected/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-kube-api-access-96dvf\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:54.107194 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.107181    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stksh\" (UniqueName: \"kubernetes.io/projected/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-kube-api-access-stksh\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:54.107194 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.107192    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:54.107194 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.107202    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:47:54.306071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.306040    2563 generic.go:356] "Generic (PLEG): container finished" podID="31f62cfe-70cc-4a3c-8c8d-b6c617a2099d" containerID="2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1" exitCode=0
Sep 09 21:47:54.306495 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.306084    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"
Sep 09 21:47:54.306495 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.306116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd" event={"ID":"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d","Type":"ContainerDied","Data":"2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1"}
Sep 09 21:47:54.306495 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.306171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd" event={"ID":"31f62cfe-70cc-4a3c-8c8d-b6c617a2099d","Type":"ContainerDied","Data":"9ec3514da855f25f1210a763d9ed9dca0aa38ca0cfc1148f73f59ab866edf679"}
Sep 09 21:47:54.306495 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.306203    2563 scope.go:117] "RemoveContainer" containerID="2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1"
Sep 09 21:47:54.307446 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.307421    2563 generic.go:356] "Generic (PLEG): container finished" podID="c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e" containerID="bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4" exitCode=0
Sep 09 21:47:54.307562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.307462    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"
Sep 09 21:47:54.307562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.307489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95" event={"ID":"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e","Type":"ContainerDied","Data":"bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4"}
Sep 09 21:47:54.307562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.307526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95" event={"ID":"c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e","Type":"ContainerDied","Data":"1bfc1f01ae79d0ce457e1bb99373a6d9e5a7d592274f509d54a3e1f8f276cb3a"}
Sep 09 21:47:54.316644 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.316622    2563 scope.go:117] "RemoveContainer" containerID="2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1"
Sep 09 21:47:54.316990 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:47:54.316963    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1\": container with ID starting with 2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1 not found: ID does not exist" containerID="2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1"
Sep 09 21:47:54.317070 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.316993    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1"} err="failed to get container status \"2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1\": rpc error: code = NotFound desc = could not find container \"2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1\": container with ID starting with 2ce9c83eeebcd480064ae4c27a611843aa476f531ed6a06617590d29924e79f1 not found: ID does not exist"
Sep 09 21:47:54.317070 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.317010    2563 scope.go:117] "RemoveContainer" containerID="bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4"
Sep 09 21:47:54.323245 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.323222    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:47:54.326177 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.326157    2563 scope.go:117] "RemoveContainer" containerID="bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4"
Sep 09 21:47:54.326539 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:47:54.326517    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4\": container with ID starting with bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4 not found: ID does not exist" containerID="bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4"
Sep 09 21:47:54.326634 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.326547    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4"} err="failed to get container status \"bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4\": rpc error: code = NotFound desc = could not find container \"bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4\": container with ID starting with bfdf05aae26bda614feac6256c8bebdd958762ee2f1fc961703b37f764e157b4 not found: ID does not exist"
Sep 09 21:47:54.347272 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.347242    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"]
Sep 09 21:47:54.354365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.354338    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b27eb4af-676599659d-zwx95"]
Sep 09 21:47:54.374658 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.374626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"]
Sep 09 21:47:54.385352 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:54.385325    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b27eb4af-7676cd8bdf-jkphd"]
Sep 09 21:47:55.046942 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:55.046909    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f62cfe-70cc-4a3c-8c8d-b6c617a2099d" path="/var/lib/kubelet/pods/31f62cfe-70cc-4a3c-8c8d-b6c617a2099d/volumes"
Sep 09 21:47:55.047285 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:55.047271    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e" path="/var/lib/kubelet/pods/c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e/volumes"
Sep 09 21:47:55.047556 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:47:55.047545    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d43cea44-3cec-4881-a842-3765f75f7030" path="/var/lib/kubelet/pods/d43cea44-3cec-4881-a842-3765f75f7030/volumes"
Sep 09 21:48:00.428914 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.428879    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"]
Sep 09 21:48:00.429364 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.429301    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e" containerName="payload-pre-processing"
Sep 09 21:48:00.429364 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.429313    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e" containerName="payload-pre-processing"
Sep 09 21:48:00.429364 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.429321    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f62cfe-70cc-4a3c-8c8d-b6c617a2099d" containerName="payload-processing"
Sep 09 21:48:00.429364 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.429330    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f62cfe-70cc-4a3c-8c8d-b6c617a2099d" containerName="payload-processing"
Sep 09 21:48:00.429542 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.429411    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f62cfe-70cc-4a3c-8c8d-b6c617a2099d" containerName="payload-processing"
Sep 09 21:48:00.429542 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.429421    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c25ad8fe-b2d9-4fc6-8bbe-9f1d5cdda00e" containerName="payload-pre-processing"
Sep 09 21:48:00.433857 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.433838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:00.438308 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.438288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ef9b2234-dockercfg-jm2w9\""
Sep 09 21:48:00.438545 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.438532    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ef9b2234\""
Sep 09 21:48:00.443036 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.443010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"]
Sep 09 21:48:00.472861 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.472826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xzs2\" (UniqueName: \"kubernetes.io/projected/08a2a46e-f586-42d1-ba75-a8134f0883ad-kube-api-access-8xzs2\") pod \"payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp\" (UID: \"08a2a46e-f586-42d1-ba75-a8134f0883ad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:00.473026 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.472984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a2a46e-f586-42d1-ba75-a8134f0883ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp\" (UID: \"08a2a46e-f586-42d1-ba75-a8134f0883ad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:00.574151 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.574117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xzs2\" (UniqueName: \"kubernetes.io/projected/08a2a46e-f586-42d1-ba75-a8134f0883ad-kube-api-access-8xzs2\") pod \"payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp\" (UID: \"08a2a46e-f586-42d1-ba75-a8134f0883ad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:00.574336 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.574206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a2a46e-f586-42d1-ba75-a8134f0883ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp\" (UID: \"08a2a46e-f586-42d1-ba75-a8134f0883ad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:00.574932 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.574913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a2a46e-f586-42d1-ba75-a8134f0883ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp\" (UID: \"08a2a46e-f586-42d1-ba75-a8134f0883ad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:00.587255 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.587220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xzs2\" (UniqueName: \"kubernetes.io/projected/08a2a46e-f586-42d1-ba75-a8134f0883ad-kube-api-access-8xzs2\") pod \"payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp\" (UID: \"08a2a46e-f586-42d1-ba75-a8134f0883ad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:00.764470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:00.764367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:01.061478 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:01.055415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"]
Sep 09 21:48:01.351642 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:01.351606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp" event={"ID":"08a2a46e-f586-42d1-ba75-a8134f0883ad","Type":"ContainerStarted","Data":"d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb"}
Sep 09 21:48:01.351853 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:01.351663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp" event={"ID":"08a2a46e-f586-42d1-ba75-a8134f0883ad","Type":"ContainerStarted","Data":"60670d37d57ad5682847be9feda7f8c5d95d6df13dec081e844d97b8a4dcbd3b"}
Sep 09 21:48:01.351853 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:01.351774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:01.369417 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:01.369369    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp" podStartSLOduration=1.369350052 podStartE2EDuration="1.369350052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:01.368962653 +0000 UTC m=+886.854845691" watchObservedRunningTime="2026-09-09 21:48:01.369350052 +0000 UTC m=+886.855233099"
Sep 09 21:48:02.476390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:02.476335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:48:02.476390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:02.476397    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:48:02.488957 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:02.488925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:48:03.373525 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:03.373492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:48:12.359976 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:12.359887    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:16.001842 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.001800    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"]
Sep 09 21:48:16.006952 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.006932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.011253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.011218    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2aaab1\"/\"kube-root-ca.crt\""
Sep 09 21:48:16.011253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.011234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2aaab1\"/\"test-model-2aaab1-kserve-self-signed-certs\""
Sep 09 21:48:16.011253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.011251    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2aaab1\"/\"openshift-service-ca.crt\""
Sep 09 21:48:16.011473 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.011294    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2aaab1\"/\"default-dockercfg-w5dn9\""
Sep 09 21:48:16.015276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.014930    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"]
Sep 09 21:48:16.138123 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.138078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tg7p\" (UniqueName: \"kubernetes.io/projected/67773263-bdc3-46b3-b6ed-ce18daebf5d1-kube-api-access-7tg7p\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.138337 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.138144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-dshm\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.138337 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.138209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-home\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.138337 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.138251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tmp-dir\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.138337 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.138305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tls-certs\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.138543 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.138438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-model-cache\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239204 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-model-cache\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tg7p\" (UniqueName: \"kubernetes.io/projected/67773263-bdc3-46b3-b6ed-ce18daebf5d1-kube-api-access-7tg7p\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-dshm\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-home\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239355 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tmp-dir\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239588 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tls-certs\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239647 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-model-cache\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239705 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-home\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.239798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.239707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tmp-dir\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.241674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.241654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-dshm\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.242106 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.242087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tls-certs\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.248682 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.248658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tg7p\" (UniqueName: \"kubernetes.io/projected/67773263-bdc3-46b3-b6ed-ce18daebf5d1-kube-api-access-7tg7p\") pod \"test-model-2aaab1-kserve-6f946d78fc-x6jld\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") " pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.318016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.317923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:16.591882 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:16.591851    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"]
Sep 09 21:48:16.593615 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:48:16.593583    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67773263_bdc3_46b3_b6ed_ce18daebf5d1.slice/crio-b593922d46a38907bb0798fa9030859257898e016cc5bb0181c1a4663352c083 WatchSource:0}: Error finding container b593922d46a38907bb0798fa9030859257898e016cc5bb0181c1a4663352c083: Status 404 returned error can't find the container with id b593922d46a38907bb0798fa9030859257898e016cc5bb0181c1a4663352c083
Sep 09 21:48:17.438131 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:17.438088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld" event={"ID":"67773263-bdc3-46b3-b6ed-ce18daebf5d1","Type":"ContainerStarted","Data":"8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93"}
Sep 09 21:48:17.438131 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:17.438136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld" event={"ID":"67773263-bdc3-46b3-b6ed-ce18daebf5d1","Type":"ContainerStarted","Data":"b593922d46a38907bb0798fa9030859257898e016cc5bb0181c1a4663352c083"}
Sep 09 21:48:17.457934 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:17.457880    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld" podStartSLOduration=2.4578645789999998 podStartE2EDuration="2.457864579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:17.455466635 +0000 UTC m=+902.941349674" watchObservedRunningTime="2026-09-09 21:48:17.457864579 +0000 UTC m=+902.943747687"
Sep 09 21:48:23.154862 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.154822    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"]
Sep 09 21:48:23.158399 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.158380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"
Sep 09 21:48:23.162834 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.162815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:48:23.182164 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.182129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"]
Sep 09 21:48:23.208091 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.208054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b86c1d-8bc1-45dc-9306-7c3d304f212f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd\" (UID: \"d2b86c1d-8bc1-45dc-9306-7c3d304f212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"
Sep 09 21:48:23.309077 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.309037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b86c1d-8bc1-45dc-9306-7c3d304f212f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd\" (UID: \"d2b86c1d-8bc1-45dc-9306-7c3d304f212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"
Sep 09 21:48:23.309740 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.309696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b86c1d-8bc1-45dc-9306-7c3d304f212f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd\" (UID: \"d2b86c1d-8bc1-45dc-9306-7c3d304f212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"
Sep 09 21:48:23.469896 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.469812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"
Sep 09 21:48:23.761697 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:23.761665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"]
Sep 09 21:48:24.476870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:24.476835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd" event={"ID":"d2b86c1d-8bc1-45dc-9306-7c3d304f212f","Type":"ContainerStarted","Data":"6eda21e83de605546368955ee50c491982540f8c3dfa0310700f4fe5b1b345eb"}
Sep 09 21:48:24.477277 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:24.476892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd" event={"ID":"d2b86c1d-8bc1-45dc-9306-7c3d304f212f","Type":"ContainerStarted","Data":"6de6d481034cab8c46bffce4f5b73889b8d3d0cf0a1f2a1f5643e77f4990781e"}
Sep 09 21:48:24.494553 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:24.494503    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd" podStartSLOduration=1.494485971 podStartE2EDuration="1.494485971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:24.493559932 +0000 UTC m=+909.979442985" watchObservedRunningTime="2026-09-09 21:48:24.494485971 +0000 UTC m=+909.980369010"
Sep 09 21:48:25.484279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:25.484240    2563 generic.go:356] "Generic (PLEG): container finished" podID="d2b86c1d-8bc1-45dc-9306-7c3d304f212f" containerID="6eda21e83de605546368955ee50c491982540f8c3dfa0310700f4fe5b1b345eb" exitCode=0
Sep 09 21:48:25.484754 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:25.484324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd" event={"ID":"d2b86c1d-8bc1-45dc-9306-7c3d304f212f","Type":"ContainerDied","Data":"6eda21e83de605546368955ee50c491982540f8c3dfa0310700f4fe5b1b345eb"}
Sep 09 21:48:26.318928 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:26.318886    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:26.319132 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:26.319037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:26.331863 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:26.331825    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:26.501260 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:26.501232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:48:26.641773 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:26.641746    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"
Sep 09 21:48:26.740301 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:26.740268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b86c1d-8bc1-45dc-9306-7c3d304f212f-maas-api-service-ca\") pod \"d2b86c1d-8bc1-45dc-9306-7c3d304f212f\" (UID: \"d2b86c1d-8bc1-45dc-9306-7c3d304f212f\") "
Sep 09 21:48:26.740618 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:26.740595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b86c1d-8bc1-45dc-9306-7c3d304f212f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2b86c1d-8bc1-45dc-9306-7c3d304f212f" (UID: "d2b86c1d-8bc1-45dc-9306-7c3d304f212f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:26.841673 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:26.841635    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b86c1d-8bc1-45dc-9306-7c3d304f212f-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:27.496652 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:27.496624    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"
Sep 09 21:48:27.496857 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:27.496659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd" event={"ID":"d2b86c1d-8bc1-45dc-9306-7c3d304f212f","Type":"ContainerDied","Data":"6de6d481034cab8c46bffce4f5b73889b8d3d0cf0a1f2a1f5643e77f4990781e"}
Sep 09 21:48:27.496857 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:27.496728    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de6d481034cab8c46bffce4f5b73889b8d3d0cf0a1f2a1f5643e77f4990781e"
Sep 09 21:48:30.505102 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.505022    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"]
Sep 09 21:48:30.507777 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.505272    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp" podUID="08a2a46e-f586-42d1-ba75-a8134f0883ad" containerName="payload-pre-processing" containerID="cri-o://d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb" gracePeriod=30
Sep 09 21:48:30.670582 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.670541    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"]
Sep 09 21:48:30.674176 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.674146    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef9b2234-kcdpd"]
Sep 09 21:48:30.770207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.770142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:30.880657 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.880622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xzs2\" (UniqueName: \"kubernetes.io/projected/08a2a46e-f586-42d1-ba75-a8134f0883ad-kube-api-access-8xzs2\") pod \"08a2a46e-f586-42d1-ba75-a8134f0883ad\" (UID: \"08a2a46e-f586-42d1-ba75-a8134f0883ad\") "
Sep 09 21:48:30.880888 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.880746    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a2a46e-f586-42d1-ba75-a8134f0883ad-plugins-config-volume\") pod \"08a2a46e-f586-42d1-ba75-a8134f0883ad\" (UID: \"08a2a46e-f586-42d1-ba75-a8134f0883ad\") "
Sep 09 21:48:30.881198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.881166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a2a46e-f586-42d1-ba75-a8134f0883ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08a2a46e-f586-42d1-ba75-a8134f0883ad" (UID: "08a2a46e-f586-42d1-ba75-a8134f0883ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:30.884118 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.884087    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a2a46e-f586-42d1-ba75-a8134f0883ad-kube-api-access-8xzs2" (OuterVolumeSpecName: "kube-api-access-8xzs2") pod "08a2a46e-f586-42d1-ba75-a8134f0883ad" (UID: "08a2a46e-f586-42d1-ba75-a8134f0883ad"). InnerVolumeSpecName "kube-api-access-8xzs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:30.981658 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.981621    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xzs2\" (UniqueName: \"kubernetes.io/projected/08a2a46e-f586-42d1-ba75-a8134f0883ad-kube-api-access-8xzs2\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:30.981658 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:30.981649    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a2a46e-f586-42d1-ba75-a8134f0883ad-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:31.047487 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.047408    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b86c1d-8bc1-45dc-9306-7c3d304f212f" path="/var/lib/kubelet/pods/d2b86c1d-8bc1-45dc-9306-7c3d304f212f/volumes"
Sep 09 21:48:31.524395 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.524353    2563 generic.go:356] "Generic (PLEG): container finished" podID="08a2a46e-f586-42d1-ba75-a8134f0883ad" containerID="d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb" exitCode=0
Sep 09 21:48:31.524916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.524409    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"
Sep 09 21:48:31.524916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.524448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp" event={"ID":"08a2a46e-f586-42d1-ba75-a8134f0883ad","Type":"ContainerDied","Data":"d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb"}
Sep 09 21:48:31.524916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.524495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp" event={"ID":"08a2a46e-f586-42d1-ba75-a8134f0883ad","Type":"ContainerDied","Data":"60670d37d57ad5682847be9feda7f8c5d95d6df13dec081e844d97b8a4dcbd3b"}
Sep 09 21:48:31.524916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.524519    2563 scope.go:117] "RemoveContainer" containerID="d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb"
Sep 09 21:48:31.533494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.533475    2563 scope.go:117] "RemoveContainer" containerID="d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb"
Sep 09 21:48:31.533820 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:48:31.533797    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb\": container with ID starting with d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb not found: ID does not exist" containerID="d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb"
Sep 09 21:48:31.533907 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.533832    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb"} err="failed to get container status \"d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb\": rpc error: code = NotFound desc = could not find container \"d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb\": container with ID starting with d486ed8b05f21f859fc87ef588f60bb5048fb7e972b5d8be34b561e773d44bcb not found: ID does not exist"
Sep 09 21:48:31.545055 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.545020    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"]
Sep 09 21:48:31.551897 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:31.551866    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef9b2234-85f8b7bf86-nh2cp"]
Sep 09 21:48:33.047178 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:33.047141    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a2a46e-f586-42d1-ba75-a8134f0883ad" path="/var/lib/kubelet/pods/08a2a46e-f586-42d1-ba75-a8134f0883ad/volumes"
Sep 09 21:48:38.397865 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:38.397828    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"]
Sep 09 21:48:53.418274 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.418214    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth" podUID="ab1a079a-5eb4-46f8-8e5c-46f64978dd32" containerName="main" containerID="cri-o://505ce3184e160bbf3770a1a0085c1f86a5541cc0df680a1ea9f26394f9a7e4b1" gracePeriod=45
Sep 09 21:48:53.644390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.644357    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab1a079a-5eb4-46f8-8e5c-46f64978dd32" containerID="505ce3184e160bbf3770a1a0085c1f86a5541cc0df680a1ea9f26394f9a7e4b1" exitCode=0
Sep 09 21:48:53.644552 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.644405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth" event={"ID":"ab1a079a-5eb4-46f8-8e5c-46f64978dd32","Type":"ContainerDied","Data":"505ce3184e160bbf3770a1a0085c1f86a5541cc0df680a1ea9f26394f9a7e4b1"}
Sep 09 21:48:53.669360 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.669296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:48:53.815766 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.815698    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-dshm\") pod \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") "
Sep 09 21:48:53.815946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.815792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snl8s\" (UniqueName: \"kubernetes.io/projected/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-kube-api-access-snl8s\") pod \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") "
Sep 09 21:48:53.815946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.815831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tls-certs\") pod \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") "
Sep 09 21:48:53.815946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.815855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-model-cache\") pod \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") "
Sep 09 21:48:53.815946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.815877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tmp-dir\") pod \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") "
Sep 09 21:48:53.815946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.815935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-home\") pod \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\" (UID: \"ab1a079a-5eb4-46f8-8e5c-46f64978dd32\") "
Sep 09 21:48:53.816201 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.816131    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab1a079a-5eb4-46f8-8e5c-46f64978dd32" (UID: "ab1a079a-5eb4-46f8-8e5c-46f64978dd32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:48:53.816201 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.816163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab1a079a-5eb4-46f8-8e5c-46f64978dd32" (UID: "ab1a079a-5eb4-46f8-8e5c-46f64978dd32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:48:53.816320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.816257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-home" (OuterVolumeSpecName: "home") pod "ab1a079a-5eb4-46f8-8e5c-46f64978dd32" (UID: "ab1a079a-5eb4-46f8-8e5c-46f64978dd32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:48:53.816367 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.816324    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:53.816367 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.816338    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:53.818237 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.818201    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab1a079a-5eb4-46f8-8e5c-46f64978dd32" (UID: "ab1a079a-5eb4-46f8-8e5c-46f64978dd32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:53.818532 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.818512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-dshm" (OuterVolumeSpecName: "dshm") pod "ab1a079a-5eb4-46f8-8e5c-46f64978dd32" (UID: "ab1a079a-5eb4-46f8-8e5c-46f64978dd32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:48:53.818595 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.818529    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-kube-api-access-snl8s" (OuterVolumeSpecName: "kube-api-access-snl8s") pod "ab1a079a-5eb4-46f8-8e5c-46f64978dd32" (UID: "ab1a079a-5eb4-46f8-8e5c-46f64978dd32"). InnerVolumeSpecName "kube-api-access-snl8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:53.916939 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.916902    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:53.916939 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.916935    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:53.916939 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.916944    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snl8s\" (UniqueName: \"kubernetes.io/projected/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-kube-api-access-snl8s\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:53.917187 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:53.916957    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1a079a-5eb4-46f8-8e5c-46f64978dd32-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:48:54.650567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:54.650534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"
Sep 09 21:48:54.650567 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:54.650553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth" event={"ID":"ab1a079a-5eb4-46f8-8e5c-46f64978dd32","Type":"ContainerDied","Data":"46d024068a41f5a7e72f6420668733b5e45a335e3badcb5b7ee0c6338559ad6b"}
Sep 09 21:48:54.651102 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:54.650603    2563 scope.go:117] "RemoveContainer" containerID="505ce3184e160bbf3770a1a0085c1f86a5541cc0df680a1ea9f26394f9a7e4b1"
Sep 09 21:48:54.672651 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:54.672616    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"]
Sep 09 21:48:54.674487 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:54.674465    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-66cedd/autores-model-66cedd-kserve-647df78cff-lpjth"]
Sep 09 21:48:55.047098 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:48:55.047007    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab1a079a-5eb4-46f8-8e5c-46f64978dd32" path="/var/lib/kubelet/pods/ab1a079a-5eb4-46f8-8e5c-46f64978dd32/volumes"
Sep 09 21:49:42.822780 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.822738    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"]
Sep 09 21:49:42.823420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823400    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab1a079a-5eb4-46f8-8e5c-46f64978dd32" containerName="main"
Sep 09 21:49:42.823469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823425    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab1a079a-5eb4-46f8-8e5c-46f64978dd32" containerName="main"
Sep 09 21:49:42.823469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823442    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a2a46e-f586-42d1-ba75-a8134f0883ad" containerName="payload-pre-processing"
Sep 09 21:49:42.823469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823452    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a2a46e-f586-42d1-ba75-a8134f0883ad" containerName="payload-pre-processing"
Sep 09 21:49:42.823562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823470    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b86c1d-8bc1-45dc-9306-7c3d304f212f" containerName="revoke-keys"
Sep 09 21:49:42.823562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823479    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b86c1d-8bc1-45dc-9306-7c3d304f212f" containerName="revoke-keys"
Sep 09 21:49:42.823639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823627    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a2a46e-f586-42d1-ba75-a8134f0883ad" containerName="payload-pre-processing"
Sep 09 21:49:42.823675 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823643    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b86c1d-8bc1-45dc-9306-7c3d304f212f" containerName="revoke-keys"
Sep 09 21:49:42.823675 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.823654    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab1a079a-5eb4-46f8-8e5c-46f64978dd32" containerName="main"
Sep 09 21:49:42.826978 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.826958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.829871 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.829846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-390abc\"/\"premium-w0-390abc-kserve-self-signed-certs\""
Sep 09 21:49:42.837913 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.837890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"]
Sep 09 21:49:42.895348 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.895307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-dshm\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.895524 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.895358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde76298-df0e-49d0-927e-cd6d114e0320-tls-certs\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.895524 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.895477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-home\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.895524 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.895514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-model-cache\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.895680 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.895541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-tmp-dir\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.895680 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.895561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlmnc\" (UniqueName: \"kubernetes.io/projected/fde76298-df0e-49d0-927e-cd6d114e0320-kube-api-access-dlmnc\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.996955 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.996912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde76298-df0e-49d0-927e-cd6d114e0320-tls-certs\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.997161 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.997020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-home\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.997161 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.997052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-model-cache\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.997161 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.997072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-tmp-dir\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.997161 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.997095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlmnc\" (UniqueName: \"kubernetes.io/projected/fde76298-df0e-49d0-927e-cd6d114e0320-kube-api-access-dlmnc\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.997161 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.997146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-dshm\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.997498 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.997474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-model-cache\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.997586 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.997524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-tmp-dir\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.997635 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.997584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-home\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.999447 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.999429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-dshm\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:42.999616 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:42.999602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde76298-df0e-49d0-927e-cd6d114e0320-tls-certs\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:43.006606 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:43.006580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlmnc\" (UniqueName: \"kubernetes.io/projected/fde76298-df0e-49d0-927e-cd6d114e0320-kube-api-access-dlmnc\") pod \"premium-w0-390abc-kserve-88d577dd-ztgnr\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") " pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:43.139242 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:43.139202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:43.411635 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:43.411602    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"]
Sep 09 21:49:43.412521 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:49:43.412496    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde76298_df0e_49d0_927e_cd6d114e0320.slice/crio-25ec7d2e6bbc5a4da7f80f08918705e070804fd8c12373ea03157b24793a1d03 WatchSource:0}: Error finding container 25ec7d2e6bbc5a4da7f80f08918705e070804fd8c12373ea03157b24793a1d03: Status 404 returned error can't find the container with id 25ec7d2e6bbc5a4da7f80f08918705e070804fd8c12373ea03157b24793a1d03
Sep 09 21:49:43.923407 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:43.923374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr" event={"ID":"fde76298-df0e-49d0-927e-cd6d114e0320","Type":"ContainerStarted","Data":"1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53"}
Sep 09 21:49:43.923813 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:43.923415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr" event={"ID":"fde76298-df0e-49d0-927e-cd6d114e0320","Type":"ContainerStarted","Data":"25ec7d2e6bbc5a4da7f80f08918705e070804fd8c12373ea03157b24793a1d03"}
Sep 09 21:49:43.942271 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:43.942211    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr" podStartSLOduration=1.942195948 podStartE2EDuration="1.942195948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:43.941100077 +0000 UTC m=+989.426983117" watchObservedRunningTime="2026-09-09 21:49:43.942195948 +0000 UTC m=+989.428078987"
Sep 09 21:49:53.139832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:53.139791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:53.140231 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:53.139847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:53.152741 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:53.152698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:49:53.990630 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:49:53.990598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:50:14.678686 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.678645    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"]
Sep 09 21:50:14.682282 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.682259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.686315 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.686288    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-da0a18\"/\"openshift-service-ca.crt\""
Sep 09 21:50:14.686442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.686348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da0a18\"/\"test-model-da0a18-kserve-self-signed-certs\""
Sep 09 21:50:14.686442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.686286    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-da0a18\"/\"kube-root-ca.crt\""
Sep 09 21:50:14.686565 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.686549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da0a18\"/\"default-dockercfg-br2t6\""
Sep 09 21:50:14.692758 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.692731    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"]
Sep 09 21:50:14.719585 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.719552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-tmp-dir\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.719832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.719595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtnm\" (UniqueName: \"kubernetes.io/projected/f20aae99-46b9-481e-bd39-b5ade73cf856-kube-api-access-zqtnm\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.719832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.719632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-home\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.719832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.719786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20aae99-46b9-481e-bd39-b5ade73cf856-tls-certs\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.720006 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.719863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-dshm\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.720006 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.719893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-model-cache\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.820774 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.820704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-dshm\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.820966 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.820780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-model-cache\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.820966 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.820869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-tmp-dir\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.820966 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.820914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtnm\" (UniqueName: \"kubernetes.io/projected/f20aae99-46b9-481e-bd39-b5ade73cf856-kube-api-access-zqtnm\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.820966 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.820951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-home\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.821187 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.821026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20aae99-46b9-481e-bd39-b5ade73cf856-tls-certs\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.821501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.821286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-model-cache\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.821501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.821353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-tmp-dir\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.821501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.821395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-home\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.823320 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.823302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-dshm\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.823617 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.823600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20aae99-46b9-481e-bd39-b5ade73cf856-tls-certs\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.828441 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.828414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtnm\" (UniqueName: \"kubernetes.io/projected/f20aae99-46b9-481e-bd39-b5ade73cf856-kube-api-access-zqtnm\") pod \"test-model-da0a18-kserve-69cc89cbf9-fkmgf\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") " pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:14.996482 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:14.996395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da0a18\"/\"default-dockercfg-br2t6\""
Sep 09 21:50:15.004420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:15.004398    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:15.283903 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:15.283876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"]
Sep 09 21:50:15.285772 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:50:15.285744    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf20aae99_46b9_481e_bd39_b5ade73cf856.slice/crio-23296a57f810509be27907bddfcc6125bab5968116449a14d1636270c2de3c4a WatchSource:0}: Error finding container 23296a57f810509be27907bddfcc6125bab5968116449a14d1636270c2de3c4a: Status 404 returned error can't find the container with id 23296a57f810509be27907bddfcc6125bab5968116449a14d1636270c2de3c4a
Sep 09 21:50:16.099308 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:16.099270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf" event={"ID":"f20aae99-46b9-481e-bd39-b5ade73cf856","Type":"ContainerStarted","Data":"30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694"}
Sep 09 21:50:16.099831 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:16.099321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf" event={"ID":"f20aae99-46b9-481e-bd39-b5ade73cf856","Type":"ContainerStarted","Data":"23296a57f810509be27907bddfcc6125bab5968116449a14d1636270c2de3c4a"}
Sep 09 21:50:16.121204 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:16.121146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf" podStartSLOduration=2.121129612 podStartE2EDuration="2.121129612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:16.119259304 +0000 UTC m=+1021.605142343" watchObservedRunningTime="2026-09-09 21:50:16.121129612 +0000 UTC m=+1021.607012650"
Sep 09 21:50:25.005124 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:25.005080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:25.005600 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:25.005362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:25.018211 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:25.018180    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:25.165513 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:25.165484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:50:44.815322 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:44.815287    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"]
Sep 09 21:50:44.818820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:44.818803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"
Sep 09 21:50:44.821500 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:44.821480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:50:44.827182 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:44.827150    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"]
Sep 09 21:50:44.919817 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:44.919769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2497da7f-6d48-4aea-96ed-b37c06a74b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4\" (UID: \"2497da7f-6d48-4aea-96ed-b37c06a74b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"
Sep 09 21:50:45.021032 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:45.020992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2497da7f-6d48-4aea-96ed-b37c06a74b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4\" (UID: \"2497da7f-6d48-4aea-96ed-b37c06a74b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"
Sep 09 21:50:45.021608 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:45.021587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2497da7f-6d48-4aea-96ed-b37c06a74b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4\" (UID: \"2497da7f-6d48-4aea-96ed-b37c06a74b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"
Sep 09 21:50:45.131706 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:45.131671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"
Sep 09 21:50:45.411089 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:45.411057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"]
Sep 09 21:50:45.413782 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:50:45.413680    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2497da7f_6d48_4aea_96ed_b37c06a74b9e.slice/crio-24124b5de3fdb2ed62a78fb945f44ff51589ee03bc20c58be404b7a2dc09a845 WatchSource:0}: Error finding container 24124b5de3fdb2ed62a78fb945f44ff51589ee03bc20c58be404b7a2dc09a845: Status 404 returned error can't find the container with id 24124b5de3fdb2ed62a78fb945f44ff51589ee03bc20c58be404b7a2dc09a845
Sep 09 21:50:46.266877 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:46.266839    2563 generic.go:356] "Generic (PLEG): container finished" podID="2497da7f-6d48-4aea-96ed-b37c06a74b9e" containerID="f1f83a955e546d16e89f79cc67ab0b55b0e2a6d0e7e662a362558e72de9c81a2" exitCode=0
Sep 09 21:50:46.267302 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:46.266924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4" event={"ID":"2497da7f-6d48-4aea-96ed-b37c06a74b9e","Type":"ContainerDied","Data":"f1f83a955e546d16e89f79cc67ab0b55b0e2a6d0e7e662a362558e72de9c81a2"}
Sep 09 21:50:46.267302 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:46.266976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4" event={"ID":"2497da7f-6d48-4aea-96ed-b37c06a74b9e","Type":"ContainerStarted","Data":"24124b5de3fdb2ed62a78fb945f44ff51589ee03bc20c58be404b7a2dc09a845"}
Sep 09 21:50:47.416803 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:47.416777    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"
Sep 09 21:50:47.548704 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:47.548604    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2497da7f-6d48-4aea-96ed-b37c06a74b9e-maas-api-service-ca\") pod \"2497da7f-6d48-4aea-96ed-b37c06a74b9e\" (UID: \"2497da7f-6d48-4aea-96ed-b37c06a74b9e\") "
Sep 09 21:50:47.549013 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:47.548988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2497da7f-6d48-4aea-96ed-b37c06a74b9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2497da7f-6d48-4aea-96ed-b37c06a74b9e" (UID: "2497da7f-6d48-4aea-96ed-b37c06a74b9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:50:47.649991 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:47.649949    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2497da7f-6d48-4aea-96ed-b37c06a74b9e-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:50:48.279682 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:48.279654    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"
Sep 09 21:50:48.279887 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:48.279689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4" event={"ID":"2497da7f-6d48-4aea-96ed-b37c06a74b9e","Type":"ContainerDied","Data":"24124b5de3fdb2ed62a78fb945f44ff51589ee03bc20c58be404b7a2dc09a845"}
Sep 09 21:50:48.279887 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:48.279762    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24124b5de3fdb2ed62a78fb945f44ff51589ee03bc20c58be404b7a2dc09a845"
Sep 09 21:50:55.411885 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:55.411840    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"]
Sep 09 21:50:55.414605 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:55.414581    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-65d42a05-kxcc4"]
Sep 09 21:50:57.046640 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:50:57.046603    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2497da7f-6d48-4aea-96ed-b37c06a74b9e" path="/var/lib/kubelet/pods/2497da7f-6d48-4aea-96ed-b37c06a74b9e/volumes"
Sep 09 21:51:35.056331 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.056242    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"]
Sep 09 21:51:35.056814 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.056696    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2497da7f-6d48-4aea-96ed-b37c06a74b9e" containerName="revoke-keys"
Sep 09 21:51:35.056814 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.056709    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2497da7f-6d48-4aea-96ed-b37c06a74b9e" containerName="revoke-keys"
Sep 09 21:51:35.056814 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.056812    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2497da7f-6d48-4aea-96ed-b37c06a74b9e" containerName="revoke-keys"
Sep 09 21:51:35.059797 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.059778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"
Sep 09 21:51:35.062803 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.062775    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:51:35.064569 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.064545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"]
Sep 09 21:51:35.124424 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.124381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6\" (UID: \"ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"
Sep 09 21:51:35.225441 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.225404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6\" (UID: \"ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"
Sep 09 21:51:35.226102 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.226083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6\" (UID: \"ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"
Sep 09 21:51:35.372845 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.372813    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"
Sep 09 21:51:35.643486 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:35.643461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"]
Sep 09 21:51:35.646327 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:51:35.646298    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8c5ef5_7dc6_4fb9_a20f_bd0350c79cb5.slice/crio-2fc707a093d46f6cbc6dfa4155b58b16c2d924646feb6ef0f0123c05b2848680 WatchSource:0}: Error finding container 2fc707a093d46f6cbc6dfa4155b58b16c2d924646feb6ef0f0123c05b2848680: Status 404 returned error can't find the container with id 2fc707a093d46f6cbc6dfa4155b58b16c2d924646feb6ef0f0123c05b2848680
Sep 09 21:51:36.544053 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:36.544021    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5" containerID="57f187660bbbfa26c964533f0e425f883e8a7b123e3984b169c6fd98da876bff" exitCode=0
Sep 09 21:51:36.544450 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:36.544090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6" event={"ID":"ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5","Type":"ContainerDied","Data":"57f187660bbbfa26c964533f0e425f883e8a7b123e3984b169c6fd98da876bff"}
Sep 09 21:51:36.544450 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:36.544127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6" event={"ID":"ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5","Type":"ContainerStarted","Data":"2fc707a093d46f6cbc6dfa4155b58b16c2d924646feb6ef0f0123c05b2848680"}
Sep 09 21:51:37.682549 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:37.682521    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"
Sep 09 21:51:37.749117 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:37.749083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5-maas-api-service-ca\") pod \"ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5\" (UID: \"ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5\") "
Sep 09 21:51:37.749482 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:37.749449    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5" (UID: "ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:51:37.850116 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:37.850078    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:51:38.562067 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:38.562018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6" event={"ID":"ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5","Type":"ContainerDied","Data":"2fc707a093d46f6cbc6dfa4155b58b16c2d924646feb6ef0f0123c05b2848680"}
Sep 09 21:51:38.562279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:38.562079    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"
Sep 09 21:51:38.562279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:38.562127    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc707a093d46f6cbc6dfa4155b58b16c2d924646feb6ef0f0123c05b2848680"
Sep 09 21:51:46.903593 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:46.903545    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"]
Sep 09 21:51:46.918891 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:46.916225    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-331c9d-qh2j6"]
Sep 09 21:51:47.046843 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:51:47.046805    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5" path="/var/lib/kubelet/pods/ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5/volumes"
Sep 09 21:52:19.882383 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.882344    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"]
Sep 09 21:52:19.882943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.882830    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5" containerName="revoke-keys"
Sep 09 21:52:19.882943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.882845    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5" containerName="revoke-keys"
Sep 09 21:52:19.883037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.882962    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea8c5ef5-7dc6-4fb9-a20f-bd0350c79cb5" containerName="revoke-keys"
Sep 09 21:52:19.886347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.886326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:19.888928 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.888904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"distinct-w4-3fa986-kserve-self-signed-certs\""
Sep 09 21:52:19.896129 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.896102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"]
Sep 09 21:52:19.971554 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.971511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlqr\" (UniqueName: \"kubernetes.io/projected/1b8eb228-1f32-49fe-9fe8-311497507a14-kube-api-access-pmlqr\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:19.971779 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.971570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-dshm\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:19.971779 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.971625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8eb228-1f32-49fe-9fe8-311497507a14-tls-certs\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:19.971779 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.971649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-model-cache\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:19.971779 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.971709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-tmp-dir\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:19.971779 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:19.971764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-home\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.072546 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.072502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlqr\" (UniqueName: \"kubernetes.io/projected/1b8eb228-1f32-49fe-9fe8-311497507a14-kube-api-access-pmlqr\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.072708 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.072567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-dshm\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.072708 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.072595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8eb228-1f32-49fe-9fe8-311497507a14-tls-certs\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.072708 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.072618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-model-cache\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.072708 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.072645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-tmp-dir\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.072708 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.072664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-home\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.073179 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.073148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-model-cache\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.073279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.073209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-home\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.073279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.073246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-tmp-dir\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.075178 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.075150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-dshm\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.075278 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.075196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8eb228-1f32-49fe-9fe8-311497507a14-tls-certs\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.080675 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.080652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlqr\" (UniqueName: \"kubernetes.io/projected/1b8eb228-1f32-49fe-9fe8-311497507a14-kube-api-access-pmlqr\") pod \"distinct-w4-3fa986-kserve-8f74d7d78-fjrbp\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") " pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.197570 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.197476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:20.471629 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.471603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"]
Sep 09 21:52:20.473850 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:52:20.473824    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b8eb228_1f32_49fe_9fe8_311497507a14.slice/crio-6dc3d77cd733e339bc6dbe1545eed035b108b118e91ac3df39e89f5924c24f31 WatchSource:0}: Error finding container 6dc3d77cd733e339bc6dbe1545eed035b108b118e91ac3df39e89f5924c24f31: Status 404 returned error can't find the container with id 6dc3d77cd733e339bc6dbe1545eed035b108b118e91ac3df39e89f5924c24f31
Sep 09 21:52:20.795041 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.794946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp" event={"ID":"1b8eb228-1f32-49fe-9fe8-311497507a14","Type":"ContainerStarted","Data":"99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2"}
Sep 09 21:52:20.795041 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.795003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp" event={"ID":"1b8eb228-1f32-49fe-9fe8-311497507a14","Type":"ContainerStarted","Data":"6dc3d77cd733e339bc6dbe1545eed035b108b118e91ac3df39e89f5924c24f31"}
Sep 09 21:52:20.811880 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:20.811823    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp" podStartSLOduration=1.811808245 podStartE2EDuration="1.811808245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:20.81000001 +0000 UTC m=+1146.295883049" watchObservedRunningTime="2026-09-09 21:52:20.811808245 +0000 UTC m=+1146.297691284"
Sep 09 21:52:30.198289 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:30.198252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:30.198695 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:30.198310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:30.210846 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:30.210812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:30.859249 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:30.859213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 21:52:42.595934 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.595894    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"]
Sep 09 21:52:42.599325 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.599306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.602377 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.602354    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-390abc\"/\"distinct-w0-390abc-kserve-self-signed-certs\""
Sep 09 21:52:42.609021 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.608706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"]
Sep 09 21:52:42.705507 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.705466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tmp-dir\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.705507 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.705512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tls-certs\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.705788 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.705537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-home\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.705788 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.705620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-dshm\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.705893 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.705796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-model-cache\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.705893 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.705874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-598dv\" (UniqueName: \"kubernetes.io/projected/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-kube-api-access-598dv\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.806911 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.806865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-dshm\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.807120 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.806992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-model-cache\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.807120 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.807060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598dv\" (UniqueName: \"kubernetes.io/projected/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-kube-api-access-598dv\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.807120 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.807095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tmp-dir\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.807282 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.807127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tls-certs\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.807282 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.807159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-home\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.807480 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.807445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-model-cache\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.807622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.807522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tmp-dir\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.807622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.807576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-home\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.809481 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.809455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-dshm\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.809733 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.809699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tls-certs\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.814800 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.814778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-598dv\" (UniqueName: \"kubernetes.io/projected/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-kube-api-access-598dv\") pod \"distinct-w0-390abc-kserve-8445b5dfc5-h4zfj\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:42.912132 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:42.912091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:43.182494 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:43.182466    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"]
Sep 09 21:52:43.183896 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:52:43.183870    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda940c2ee_0fc5_4d46_9d97_0689d9f10ecc.slice/crio-717050099d27c23b39d4e92550e82ffda981c47225eb5900681c921ac35dfd0b WatchSource:0}: Error finding container 717050099d27c23b39d4e92550e82ffda981c47225eb5900681c921ac35dfd0b: Status 404 returned error can't find the container with id 717050099d27c23b39d4e92550e82ffda981c47225eb5900681c921ac35dfd0b
Sep 09 21:52:43.920267 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:43.920230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj" event={"ID":"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc","Type":"ContainerStarted","Data":"7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0"}
Sep 09 21:52:43.920267 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:43.920274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj" event={"ID":"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc","Type":"ContainerStarted","Data":"717050099d27c23b39d4e92550e82ffda981c47225eb5900681c921ac35dfd0b"}
Sep 09 21:52:43.938281 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:43.938220    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj" podStartSLOduration=1.938205606 podStartE2EDuration="1.938205606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:43.935515086 +0000 UTC m=+1169.421398127" watchObservedRunningTime="2026-09-09 21:52:43.938205606 +0000 UTC m=+1169.424088644"
Sep 09 21:52:52.913267 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:52.913230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:52.913845 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:52.913287    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:52.926293 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:52.926269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:52.983686 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:52.983658    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:52:58.771636 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:52:58.771590    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"]
Sep 09 21:53:13.793411 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:13.793348    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld" podUID="67773263-bdc3-46b3-b6ed-ce18daebf5d1" containerName="main" containerID="cri-o://8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93" gracePeriod=45
Sep 09 21:53:14.061280 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.061257    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:53:14.096233 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.096201    2563 generic.go:356] "Generic (PLEG): container finished" podID="67773263-bdc3-46b3-b6ed-ce18daebf5d1" containerID="8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93" exitCode=0
Sep 09 21:53:14.096424 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.096242    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"
Sep 09 21:53:14.096424 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.096291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld" event={"ID":"67773263-bdc3-46b3-b6ed-ce18daebf5d1","Type":"ContainerDied","Data":"8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93"}
Sep 09 21:53:14.096424 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.096333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld" event={"ID":"67773263-bdc3-46b3-b6ed-ce18daebf5d1","Type":"ContainerDied","Data":"b593922d46a38907bb0798fa9030859257898e016cc5bb0181c1a4663352c083"}
Sep 09 21:53:14.096424 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.096354    2563 scope.go:117] "RemoveContainer" containerID="8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93"
Sep 09 21:53:14.107511 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.107493    2563 scope.go:117] "RemoveContainer" containerID="8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93"
Sep 09 21:53:14.107791 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:53:14.107772    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93\": container with ID starting with 8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93 not found: ID does not exist" containerID="8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93"
Sep 09 21:53:14.107844 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.107806    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93"} err="failed to get container status \"8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93\": rpc error: code = NotFound desc = could not find container \"8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93\": container with ID starting with 8d91d9eb3c0c6c0d601590246a46acac48a4c15661f6e7383179e57f2b1dfa93 not found: ID does not exist"
Sep 09 21:53:14.237712 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.237678    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-model-cache\") pod \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") "
Sep 09 21:53:14.237878 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.237758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tg7p\" (UniqueName: \"kubernetes.io/projected/67773263-bdc3-46b3-b6ed-ce18daebf5d1-kube-api-access-7tg7p\") pod \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") "
Sep 09 21:53:14.237878 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.237862    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tls-certs\") pod \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") "
Sep 09 21:53:14.237968 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.237894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-dshm\") pod \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") "
Sep 09 21:53:14.237968 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.237938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-home\") pod \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") "
Sep 09 21:53:14.238058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.237989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tmp-dir\") pod \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\" (UID: \"67773263-bdc3-46b3-b6ed-ce18daebf5d1\") "
Sep 09 21:53:14.238112 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.237988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "67773263-bdc3-46b3-b6ed-ce18daebf5d1" (UID: "67773263-bdc3-46b3-b6ed-ce18daebf5d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:53:14.238470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.238192    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-home" (OuterVolumeSpecName: "home") pod "67773263-bdc3-46b3-b6ed-ce18daebf5d1" (UID: "67773263-bdc3-46b3-b6ed-ce18daebf5d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:53:14.238470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.238239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67773263-bdc3-46b3-b6ed-ce18daebf5d1" (UID: "67773263-bdc3-46b3-b6ed-ce18daebf5d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:53:14.238620 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.238504    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:14.238620 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.238524    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:14.238620 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.238558    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:14.240312 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.240292    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67773263-bdc3-46b3-b6ed-ce18daebf5d1-kube-api-access-7tg7p" (OuterVolumeSpecName: "kube-api-access-7tg7p") pod "67773263-bdc3-46b3-b6ed-ce18daebf5d1" (UID: "67773263-bdc3-46b3-b6ed-ce18daebf5d1"). InnerVolumeSpecName "kube-api-access-7tg7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:53:14.240680 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.240660    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-dshm" (OuterVolumeSpecName: "dshm") pod "67773263-bdc3-46b3-b6ed-ce18daebf5d1" (UID: "67773263-bdc3-46b3-b6ed-ce18daebf5d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:53:14.240680 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.240668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67773263-bdc3-46b3-b6ed-ce18daebf5d1" (UID: "67773263-bdc3-46b3-b6ed-ce18daebf5d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:53:14.340115 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.340029    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tg7p\" (UniqueName: \"kubernetes.io/projected/67773263-bdc3-46b3-b6ed-ce18daebf5d1-kube-api-access-7tg7p\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:14.340115 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.340063    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67773263-bdc3-46b3-b6ed-ce18daebf5d1-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:14.340115 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.340072    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67773263-bdc3-46b3-b6ed-ce18daebf5d1-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:14.418517 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.418485    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"]
Sep 09 21:53:14.422148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:14.422123    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2aaab1/test-model-2aaab1-kserve-6f946d78fc-x6jld"]
Sep 09 21:53:15.050670 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:15.050631    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67773263-bdc3-46b3-b6ed-ce18daebf5d1" path="/var/lib/kubelet/pods/67773263-bdc3-46b3-b6ed-ce18daebf5d1/volumes"
Sep 09 21:53:15.106277 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:15.106244    2563 scope.go:117] "RemoveContainer" containerID="76e1d60e25a4e6a5c9d4d6604e6d70de8f1817a8f1b1caef6d6ce39477d26cf8"
Sep 09 21:53:15.120298 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:15.120259    2563 scope.go:117] "RemoveContainer" containerID="9f1828a7511ba4729061fdf9ea57c8648289504fb8d471306955739142b250d9"
Sep 09 21:53:21.601691 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.601655    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"]
Sep 09 21:53:21.602310 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.602290    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67773263-bdc3-46b3-b6ed-ce18daebf5d1" containerName="main"
Sep 09 21:53:21.602393 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.602314    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="67773263-bdc3-46b3-b6ed-ce18daebf5d1" containerName="main"
Sep 09 21:53:21.602477 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.602465    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="67773263-bdc3-46b3-b6ed-ce18daebf5d1" containerName="main"
Sep 09 21:53:21.612386 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.612356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.615572 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.615547    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4883373b\"/\"openshift-service-ca.crt\""
Sep 09 21:53:21.615750 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.615553    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4883373b\"/\"default-dockercfg-kndfm\""
Sep 09 21:53:21.615962 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.615943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"]
Sep 09 21:53:21.616888 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.616872    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4883373b\"/\"kube-root-ca.crt\""
Sep 09 21:53:21.617206 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.617189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4883373b\"/\"e2e-lifecycle-model-4883373b-kserve-self-signed-certs\""
Sep 09 21:53:21.716415 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.716378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9h2s\" (UniqueName: \"kubernetes.io/projected/bd582094-39bc-40a7-a823-a7fd6da4229f-kube-api-access-v9h2s\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.716415 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.716418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-dshm\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.716654 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.716435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-model-cache\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.716654 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.716463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-tmp-dir\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.716654 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.716523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582094-39bc-40a7-a823-a7fd6da4229f-tls-certs\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.716654 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.716578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-home\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.817451 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.817414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-tmp-dir\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.817639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.817473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582094-39bc-40a7-a823-a7fd6da4229f-tls-certs\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.817639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.817505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-home\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.817639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.817598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9h2s\" (UniqueName: \"kubernetes.io/projected/bd582094-39bc-40a7-a823-a7fd6da4229f-kube-api-access-v9h2s\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.817639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.817614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-dshm\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.817639 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.817631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-model-cache\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.817946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.817867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-tmp-dir\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.818016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.817997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-home\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.818093 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.818063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-model-cache\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.820053 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.820029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-dshm\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.820199 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.820180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582094-39bc-40a7-a823-a7fd6da4229f-tls-certs\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.827922 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.827904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9h2s\" (UniqueName: \"kubernetes.io/projected/bd582094-39bc-40a7-a823-a7fd6da4229f-kube-api-access-v9h2s\") pod \"e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") " pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:21.923668 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:21.923548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:22.197953 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:22.197918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"]
Sep 09 21:53:22.199387 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:53:22.199358    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd582094_39bc_40a7_a823_a7fd6da4229f.slice/crio-cac880ca174671f4e511c93a2421faa4dce69738ec6cc38fb4abcb90fe712c92 WatchSource:0}: Error finding container cac880ca174671f4e511c93a2421faa4dce69738ec6cc38fb4abcb90fe712c92: Status 404 returned error can't find the container with id cac880ca174671f4e511c93a2421faa4dce69738ec6cc38fb4abcb90fe712c92
Sep 09 21:53:23.153679 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:23.153634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m" event={"ID":"bd582094-39bc-40a7-a823-a7fd6da4229f","Type":"ContainerStarted","Data":"b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2"}
Sep 09 21:53:23.154138 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:23.153685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m" event={"ID":"bd582094-39bc-40a7-a823-a7fd6da4229f","Type":"ContainerStarted","Data":"cac880ca174671f4e511c93a2421faa4dce69738ec6cc38fb4abcb90fe712c92"}
Sep 09 21:53:23.171825 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:23.171778    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m" podStartSLOduration=2.171761636 podStartE2EDuration="2.171761636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:53:23.16976435 +0000 UTC m=+1208.655647388" watchObservedRunningTime="2026-09-09 21:53:23.171761636 +0000 UTC m=+1208.657644674"
Sep 09 21:53:28.038980 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.038942    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"]
Sep 09 21:53:28.042403 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.042380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"
Sep 09 21:53:28.045420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.045398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:53:28.050517 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.050490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"]
Sep 09 21:53:28.185399 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.185256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/145be676-c205-4d0b-98cd-65be39144279-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn\" (UID: \"145be676-c205-4d0b-98cd-65be39144279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"
Sep 09 21:53:28.286613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.286576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/145be676-c205-4d0b-98cd-65be39144279-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn\" (UID: \"145be676-c205-4d0b-98cd-65be39144279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"
Sep 09 21:53:28.287240 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.287218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/145be676-c205-4d0b-98cd-65be39144279-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn\" (UID: \"145be676-c205-4d0b-98cd-65be39144279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"
Sep 09 21:53:28.355770 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.355743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"
Sep 09 21:53:28.626933 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:28.626902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"]
Sep 09 21:53:28.628747 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:53:28.628696    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145be676_c205_4d0b_98cd_65be39144279.slice/crio-df97620002f4c43ab8c10e4d20ab0aa7d83fc25be3605d0965287722db39f21d WatchSource:0}: Error finding container df97620002f4c43ab8c10e4d20ab0aa7d83fc25be3605d0965287722db39f21d: Status 404 returned error can't find the container with id df97620002f4c43ab8c10e4d20ab0aa7d83fc25be3605d0965287722db39f21d
Sep 09 21:53:29.191948 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:29.191914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn" event={"ID":"145be676-c205-4d0b-98cd-65be39144279","Type":"ContainerStarted","Data":"e2cfdbb15e48d07e55c0529a34c45fe132090c5b706f997603f87ebeb464e2b6"}
Sep 09 21:53:29.192332 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:29.191969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn" event={"ID":"145be676-c205-4d0b-98cd-65be39144279","Type":"ContainerStarted","Data":"df97620002f4c43ab8c10e4d20ab0aa7d83fc25be3605d0965287722db39f21d"}
Sep 09 21:53:29.207525 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:29.207478    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn" podStartSLOduration=1.207460752 podStartE2EDuration="1.207460752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:53:29.206071215 +0000 UTC m=+1214.691954252" watchObservedRunningTime="2026-09-09 21:53:29.207460752 +0000 UTC m=+1214.693343790"
Sep 09 21:53:30.202207 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:30.202163    2563 generic.go:356] "Generic (PLEG): container finished" podID="145be676-c205-4d0b-98cd-65be39144279" containerID="e2cfdbb15e48d07e55c0529a34c45fe132090c5b706f997603f87ebeb464e2b6" exitCode=0
Sep 09 21:53:30.202692 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:30.202352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn" event={"ID":"145be676-c205-4d0b-98cd-65be39144279","Type":"ContainerDied","Data":"e2cfdbb15e48d07e55c0529a34c45fe132090c5b706f997603f87ebeb464e2b6"}
Sep 09 21:53:31.349913 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:31.349887    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"
Sep 09 21:53:31.418859 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:31.418818    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/145be676-c205-4d0b-98cd-65be39144279-maas-api-service-ca\") pod \"145be676-c205-4d0b-98cd-65be39144279\" (UID: \"145be676-c205-4d0b-98cd-65be39144279\") "
Sep 09 21:53:31.419192 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:31.419169    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145be676-c205-4d0b-98cd-65be39144279-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "145be676-c205-4d0b-98cd-65be39144279" (UID: "145be676-c205-4d0b-98cd-65be39144279"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:53:31.520291 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:31.520209    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/145be676-c205-4d0b-98cd-65be39144279-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:31.924395 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:31.924353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:31.924577 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:31.924416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:31.937445 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:31.937414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:32.214347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:32.214257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn" event={"ID":"145be676-c205-4d0b-98cd-65be39144279","Type":"ContainerDied","Data":"df97620002f4c43ab8c10e4d20ab0aa7d83fc25be3605d0965287722db39f21d"}
Sep 09 21:53:32.214347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:32.214314    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df97620002f4c43ab8c10e4d20ab0aa7d83fc25be3605d0965287722db39f21d"
Sep 09 21:53:32.214347 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:32.214273    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"
Sep 09 21:53:32.226992 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:32.226965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:53:38.525164 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.525130    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"]
Sep 09 21:53:38.525608 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.525371    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg" podUID="ff732478-a16b-4998-b61a-113bd9cc83a0" containerName="payload-processing" containerID="cri-o://a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b" gracePeriod=30
Sep 09 21:53:38.535558 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.534697    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"]
Sep 09 21:53:38.535558 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.535391    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g" podUID="cd9ab66b-bef8-4d34-b0d3-f7124d2e395c" containerName="payload-pre-processing" containerID="cri-o://b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724" gracePeriod=30
Sep 09 21:53:38.658894 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:53:38.658860    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9ab66b_bef8_4d34_b0d3_f7124d2e395c.slice/crio-conmon-b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9ab66b_bef8_4d34_b0d3_f7124d2e395c.slice/crio-b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 21:53:38.690142 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.690097    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"]
Sep 09 21:53:38.692522 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.692493    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2aaab1-jghqn"]
Sep 09 21:53:38.818343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.818317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:53:38.821594 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.821571    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:53:38.896566 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.896530    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff732478-a16b-4998-b61a-113bd9cc83a0-plugins-config-volume\") pod \"ff732478-a16b-4998-b61a-113bd9cc83a0\" (UID: \"ff732478-a16b-4998-b61a-113bd9cc83a0\") "
Sep 09 21:53:38.896765 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.896652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4nrs\" (UniqueName: \"kubernetes.io/projected/ff732478-a16b-4998-b61a-113bd9cc83a0-kube-api-access-z4nrs\") pod \"ff732478-a16b-4998-b61a-113bd9cc83a0\" (UID: \"ff732478-a16b-4998-b61a-113bd9cc83a0\") "
Sep 09 21:53:38.897017 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.896994    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff732478-a16b-4998-b61a-113bd9cc83a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff732478-a16b-4998-b61a-113bd9cc83a0" (UID: "ff732478-a16b-4998-b61a-113bd9cc83a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:53:38.898849 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.898824    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff732478-a16b-4998-b61a-113bd9cc83a0-kube-api-access-z4nrs" (OuterVolumeSpecName: "kube-api-access-z4nrs") pod "ff732478-a16b-4998-b61a-113bd9cc83a0" (UID: "ff732478-a16b-4998-b61a-113bd9cc83a0"). InnerVolumeSpecName "kube-api-access-z4nrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:53:38.997641 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.997607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-plugins-config-volume\") pod \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\" (UID: \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\") "
Sep 09 21:53:38.997829 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.997689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tclzw\" (UniqueName: \"kubernetes.io/projected/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-kube-api-access-tclzw\") pod \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\" (UID: \"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c\") "
Sep 09 21:53:38.998031 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.998013    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff732478-a16b-4998-b61a-113bd9cc83a0-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:38.998068 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.998037    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nrs\" (UniqueName: \"kubernetes.io/projected/ff732478-a16b-4998-b61a-113bd9cc83a0-kube-api-access-z4nrs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:38.998104 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.998071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd9ab66b-bef8-4d34-b0d3-f7124d2e395c" (UID: "cd9ab66b-bef8-4d34-b0d3-f7124d2e395c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:53:38.999922 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:38.999902    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-kube-api-access-tclzw" (OuterVolumeSpecName: "kube-api-access-tclzw") pod "cd9ab66b-bef8-4d34-b0d3-f7124d2e395c" (UID: "cd9ab66b-bef8-4d34-b0d3-f7124d2e395c"). InnerVolumeSpecName "kube-api-access-tclzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:53:39.048573 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.048489    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145be676-c205-4d0b-98cd-65be39144279" path="/var/lib/kubelet/pods/145be676-c205-4d0b-98cd-65be39144279/volumes"
Sep 09 21:53:39.098776 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.098750    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:39.098776 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.098775    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tclzw\" (UniqueName: \"kubernetes.io/projected/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c-kube-api-access-tclzw\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:39.258067 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.258033    2563 generic.go:356] "Generic (PLEG): container finished" podID="ff732478-a16b-4998-b61a-113bd9cc83a0" containerID="a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b" exitCode=0
Sep 09 21:53:39.258267 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.258088    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"
Sep 09 21:53:39.258267 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.258133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg" event={"ID":"ff732478-a16b-4998-b61a-113bd9cc83a0","Type":"ContainerDied","Data":"a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b"}
Sep 09 21:53:39.258267 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.258201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg" event={"ID":"ff732478-a16b-4998-b61a-113bd9cc83a0","Type":"ContainerDied","Data":"b8ad5f108fd7f48e13be23f198e1ef97e408917c7a7adfc7fbfc677da0f19484"}
Sep 09 21:53:39.258267 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.258229    2563 scope.go:117] "RemoveContainer" containerID="a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b"
Sep 09 21:53:39.259485 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.259461    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd9ab66b-bef8-4d34-b0d3-f7124d2e395c" containerID="b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724" exitCode=0
Sep 09 21:53:39.259604 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.259492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g" event={"ID":"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c","Type":"ContainerDied","Data":"b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724"}
Sep 09 21:53:39.259604 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.259558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g" event={"ID":"cd9ab66b-bef8-4d34-b0d3-f7124d2e395c","Type":"ContainerDied","Data":"d526f47efe0ad75718bfc95e238cf556313d7291b007b0f78facaa6fd7c2def6"}
Sep 09 21:53:39.259604 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.259508    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"
Sep 09 21:53:39.267589 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.267572    2563 scope.go:117] "RemoveContainer" containerID="a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b"
Sep 09 21:53:39.267955 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:53:39.267934    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b\": container with ID starting with a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b not found: ID does not exist" containerID="a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b"
Sep 09 21:53:39.268049 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.267966    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b"} err="failed to get container status \"a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b\": rpc error: code = NotFound desc = could not find container \"a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b\": container with ID starting with a9d3a3dc3cfcb3bb291ba35f2fdf4ccf455eb48cb30de019491aaae0c5b4c55b not found: ID does not exist"
Sep 09 21:53:39.268049 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.267986    2563 scope.go:117] "RemoveContainer" containerID="b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724"
Sep 09 21:53:39.275362 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.275341    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"]
Sep 09 21:53:39.276691 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.276672    2563 scope.go:117] "RemoveContainer" containerID="b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724"
Sep 09 21:53:39.277056 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:53:39.277029    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724\": container with ID starting with b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724 not found: ID does not exist" containerID="b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724"
Sep 09 21:53:39.277158 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.277066    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724"} err="failed to get container status \"b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724\": rpc error: code = NotFound desc = could not find container \"b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724\": container with ID starting with b6adaa85338e606e098db88376ba24f5743ea416e1a459cc1ddcc282712ff724 not found: ID does not exist"
Sep 09 21:53:39.279350 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.279332    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2aaab1-68b5dd8b7c-xqvkg"]
Sep 09 21:53:39.293087 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.293066    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"]
Sep 09 21:53:39.294886 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:39.294868    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2aaab1-c64d77d5d-9vv8g"]
Sep 09 21:53:41.055338 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:41.055298    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9ab66b-bef8-4d34-b0d3-f7124d2e395c" path="/var/lib/kubelet/pods/cd9ab66b-bef8-4d34-b0d3-f7124d2e395c/volumes"
Sep 09 21:53:41.055754 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:41.055678    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff732478-a16b-4998-b61a-113bd9cc83a0" path="/var/lib/kubelet/pods/ff732478-a16b-4998-b61a-113bd9cc83a0/volumes"
Sep 09 21:53:45.438099 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438062    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"]
Sep 09 21:53:45.438551 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145be676-c205-4d0b-98cd-65be39144279" containerName="revoke-keys"
Sep 09 21:53:45.438551 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438480    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="145be676-c205-4d0b-98cd-65be39144279" containerName="revoke-keys"
Sep 09 21:53:45.438551 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438505    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff732478-a16b-4998-b61a-113bd9cc83a0" containerName="payload-processing"
Sep 09 21:53:45.438551 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438511    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff732478-a16b-4998-b61a-113bd9cc83a0" containerName="payload-processing"
Sep 09 21:53:45.438551 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438519    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9ab66b-bef8-4d34-b0d3-f7124d2e395c" containerName="payload-pre-processing"
Sep 09 21:53:45.438551 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438525    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9ab66b-bef8-4d34-b0d3-f7124d2e395c" containerName="payload-pre-processing"
Sep 09 21:53:45.438767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438598    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9ab66b-bef8-4d34-b0d3-f7124d2e395c" containerName="payload-pre-processing"
Sep 09 21:53:45.438767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438610    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff732478-a16b-4998-b61a-113bd9cc83a0" containerName="payload-processing"
Sep 09 21:53:45.438767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.438619    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="145be676-c205-4d0b-98cd-65be39144279" containerName="revoke-keys"
Sep 09 21:53:45.441767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.441751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"
Sep 09 21:53:45.445018 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.444995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:53:45.452923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.452902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"]
Sep 09 21:53:45.466091 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.466063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64ef116-7daa-430f-b4c4-e83a10369c02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b\" (UID: \"d64ef116-7daa-430f-b4c4-e83a10369c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"
Sep 09 21:53:45.567253 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.567213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64ef116-7daa-430f-b4c4-e83a10369c02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b\" (UID: \"d64ef116-7daa-430f-b4c4-e83a10369c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"
Sep 09 21:53:45.567897 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.567875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64ef116-7daa-430f-b4c4-e83a10369c02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b\" (UID: \"d64ef116-7daa-430f-b4c4-e83a10369c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"
Sep 09 21:53:45.753432 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:45.753349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"
Sep 09 21:53:46.026441 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:46.026408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"]
Sep 09 21:53:46.028260 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:53:46.028223    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64ef116_7daa_430f_b4c4_e83a10369c02.slice/crio-b892e68d290001f3909c5a4daa9048a6623282b4ac0eb4a77daad9b46266c465 WatchSource:0}: Error finding container b892e68d290001f3909c5a4daa9048a6623282b4ac0eb4a77daad9b46266c465: Status 404 returned error can't find the container with id b892e68d290001f3909c5a4daa9048a6623282b4ac0eb4a77daad9b46266c465
Sep 09 21:53:46.302674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:46.302631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b" event={"ID":"d64ef116-7daa-430f-b4c4-e83a10369c02","Type":"ContainerStarted","Data":"b892e68d290001f3909c5a4daa9048a6623282b4ac0eb4a77daad9b46266c465"}
Sep 09 21:53:47.317422 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:47.317384    2563 generic.go:356] "Generic (PLEG): container finished" podID="d64ef116-7daa-430f-b4c4-e83a10369c02" containerID="00963a416b187765c1c9822ddb38f5e22053dcff63603fa1ab2b1e5ccd146e3e" exitCode=0
Sep 09 21:53:47.317834 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:47.317462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b" event={"ID":"d64ef116-7daa-430f-b4c4-e83a10369c02","Type":"ContainerDied","Data":"00963a416b187765c1c9822ddb38f5e22053dcff63603fa1ab2b1e5ccd146e3e"}
Sep 09 21:53:48.464694 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:48.464661    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"
Sep 09 21:53:48.498309 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:48.498277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64ef116-7daa-430f-b4c4-e83a10369c02-maas-api-service-ca\") pod \"d64ef116-7daa-430f-b4c4-e83a10369c02\" (UID: \"d64ef116-7daa-430f-b4c4-e83a10369c02\") "
Sep 09 21:53:48.498631 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:48.498606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64ef116-7daa-430f-b4c4-e83a10369c02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d64ef116-7daa-430f-b4c4-e83a10369c02" (UID: "d64ef116-7daa-430f-b4c4-e83a10369c02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:53:48.599219 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:48.599181    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64ef116-7daa-430f-b4c4-e83a10369c02-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:49.330016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:49.329936    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"
Sep 09 21:53:49.330016 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:49.329977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b" event={"ID":"d64ef116-7daa-430f-b4c4-e83a10369c02","Type":"ContainerDied","Data":"b892e68d290001f3909c5a4daa9048a6623282b4ac0eb4a77daad9b46266c465"}
Sep 09 21:53:49.330200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:49.330031    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b892e68d290001f3909c5a4daa9048a6623282b4ac0eb4a77daad9b46266c465"
Sep 09 21:53:55.909819 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:55.909785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"]
Sep 09 21:53:55.910288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:55.910020    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl" podUID="afc84ea3-4140-4e93-8dbe-01eec364d55f" containerName="payload-processing" containerID="cri-o://1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83" gracePeriod=30
Sep 09 21:53:55.917020 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:55.916420    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"]
Sep 09 21:53:55.917020 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:55.916648    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw" podUID="33afc968-7590-4088-b21b-73b69e6008fc" containerName="payload-pre-processing" containerID="cri-o://a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43" gracePeriod=30
Sep 09 21:53:56.079536 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.079458    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"]
Sep 09 21:53:56.082564 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.082534    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da0a18-57s5b"]
Sep 09 21:53:56.188261 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.188228    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:53:56.208107 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.208080    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:53:56.278804 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.278771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqjdl\" (UniqueName: \"kubernetes.io/projected/afc84ea3-4140-4e93-8dbe-01eec364d55f-kube-api-access-cqjdl\") pod \"afc84ea3-4140-4e93-8dbe-01eec364d55f\" (UID: \"afc84ea3-4140-4e93-8dbe-01eec364d55f\") "
Sep 09 21:53:56.278971 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.278842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33afc968-7590-4088-b21b-73b69e6008fc-plugins-config-volume\") pod \"33afc968-7590-4088-b21b-73b69e6008fc\" (UID: \"33afc968-7590-4088-b21b-73b69e6008fc\") "
Sep 09 21:53:56.278971 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.278931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc84ea3-4140-4e93-8dbe-01eec364d55f-plugins-config-volume\") pod \"afc84ea3-4140-4e93-8dbe-01eec364d55f\" (UID: \"afc84ea3-4140-4e93-8dbe-01eec364d55f\") "
Sep 09 21:53:56.278971 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.278956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqr2s\" (UniqueName: \"kubernetes.io/projected/33afc968-7590-4088-b21b-73b69e6008fc-kube-api-access-jqr2s\") pod \"33afc968-7590-4088-b21b-73b69e6008fc\" (UID: \"33afc968-7590-4088-b21b-73b69e6008fc\") "
Sep 09 21:53:56.279264 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.279225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33afc968-7590-4088-b21b-73b69e6008fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33afc968-7590-4088-b21b-73b69e6008fc" (UID: "33afc968-7590-4088-b21b-73b69e6008fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:53:56.279393 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.279289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc84ea3-4140-4e93-8dbe-01eec364d55f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afc84ea3-4140-4e93-8dbe-01eec364d55f" (UID: "afc84ea3-4140-4e93-8dbe-01eec364d55f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:53:56.281081 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.281059    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc84ea3-4140-4e93-8dbe-01eec364d55f-kube-api-access-cqjdl" (OuterVolumeSpecName: "kube-api-access-cqjdl") pod "afc84ea3-4140-4e93-8dbe-01eec364d55f" (UID: "afc84ea3-4140-4e93-8dbe-01eec364d55f"). InnerVolumeSpecName "kube-api-access-cqjdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:53:56.281167 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.281119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33afc968-7590-4088-b21b-73b69e6008fc-kube-api-access-jqr2s" (OuterVolumeSpecName: "kube-api-access-jqr2s") pod "33afc968-7590-4088-b21b-73b69e6008fc" (UID: "33afc968-7590-4088-b21b-73b69e6008fc"). InnerVolumeSpecName "kube-api-access-jqr2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:53:56.371619 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.371579    2563 generic.go:356] "Generic (PLEG): container finished" podID="33afc968-7590-4088-b21b-73b69e6008fc" containerID="a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43" exitCode=0
Sep 09 21:53:56.371819 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.371624    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"
Sep 09 21:53:56.371819 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.371668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw" event={"ID":"33afc968-7590-4088-b21b-73b69e6008fc","Type":"ContainerDied","Data":"a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43"}
Sep 09 21:53:56.371819 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.371737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw" event={"ID":"33afc968-7590-4088-b21b-73b69e6008fc","Type":"ContainerDied","Data":"b8284c38ab7a1ab3634f767ae20bb0be403c1cc055d1de5a5f7454d217e9b9fc"}
Sep 09 21:53:56.371819 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.371771    2563 scope.go:117] "RemoveContainer" containerID="a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43"
Sep 09 21:53:56.373204 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.373180    2563 generic.go:356] "Generic (PLEG): container finished" podID="afc84ea3-4140-4e93-8dbe-01eec364d55f" containerID="1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83" exitCode=0
Sep 09 21:53:56.373506 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.373222    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"
Sep 09 21:53:56.373506 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.373258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl" event={"ID":"afc84ea3-4140-4e93-8dbe-01eec364d55f","Type":"ContainerDied","Data":"1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83"}
Sep 09 21:53:56.373506 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.373301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl" event={"ID":"afc84ea3-4140-4e93-8dbe-01eec364d55f","Type":"ContainerDied","Data":"c6553aff08291d2480259fa06875386db36c46f86b72d39b0f1e44257a07283d"}
Sep 09 21:53:56.377791 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.377769    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"]
Sep 09 21:53:56.378352 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378333    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33afc968-7590-4088-b21b-73b69e6008fc" containerName="payload-pre-processing"
Sep 09 21:53:56.378453 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="33afc968-7590-4088-b21b-73b69e6008fc" containerName="payload-pre-processing"
Sep 09 21:53:56.378453 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378389    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64ef116-7daa-430f-b4c4-e83a10369c02" containerName="revoke-keys"
Sep 09 21:53:56.378453 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378400    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64ef116-7daa-430f-b4c4-e83a10369c02" containerName="revoke-keys"
Sep 09 21:53:56.378453 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378426    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc84ea3-4140-4e93-8dbe-01eec364d55f" containerName="payload-processing"
Sep 09 21:53:56.378453 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378434    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc84ea3-4140-4e93-8dbe-01eec364d55f" containerName="payload-processing"
Sep 09 21:53:56.378709 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378548    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64ef116-7daa-430f-b4c4-e83a10369c02" containerName="revoke-keys"
Sep 09 21:53:56.378709 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378560    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="33afc968-7590-4088-b21b-73b69e6008fc" containerName="payload-pre-processing"
Sep 09 21:53:56.378709 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.378574    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc84ea3-4140-4e93-8dbe-01eec364d55f" containerName="payload-processing"
Sep 09 21:53:56.379828 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.379802    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33afc968-7590-4088-b21b-73b69e6008fc-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:56.379936 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.379833    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc84ea3-4140-4e93-8dbe-01eec364d55f-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:56.379936 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.379849    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqr2s\" (UniqueName: \"kubernetes.io/projected/33afc968-7590-4088-b21b-73b69e6008fc-kube-api-access-jqr2s\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:56.379936 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.379859    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqjdl\" (UniqueName: \"kubernetes.io/projected/afc84ea3-4140-4e93-8dbe-01eec364d55f-kube-api-access-cqjdl\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:53:56.382501 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.382480    2563 scope.go:117] "RemoveContainer" containerID="a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43"
Sep 09 21:53:56.382853 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:53:56.382835    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43\": container with ID starting with a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43 not found: ID does not exist" containerID="a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43"
Sep 09 21:53:56.382920 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.382860    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43"} err="failed to get container status \"a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43\": rpc error: code = NotFound desc = could not find container \"a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43\": container with ID starting with a07f8273e532b56b098277d08138c8ac703bb6545053649f39c163c0c7c9ff43 not found: ID does not exist"
Sep 09 21:53:56.382920 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.382877    2563 scope.go:117] "RemoveContainer" containerID="1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83"
Sep 09 21:53:56.383521 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.383505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.386322 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.386302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-749a53\"/\"default-dockercfg-5khx6\""
Sep 09 21:53:56.386439 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.386302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-749a53\"/\"noten-model-749a53-kserve-self-signed-certs\""
Sep 09 21:53:56.386439 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.386348    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-749a53\"/\"kube-root-ca.crt\""
Sep 09 21:53:56.386439 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.386351    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-749a53\"/\"openshift-service-ca.crt\""
Sep 09 21:53:56.392128 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.392103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"]
Sep 09 21:53:56.394767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.394746    2563 scope.go:117] "RemoveContainer" containerID="1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83"
Sep 09 21:53:56.395115 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:53:56.395094    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83\": container with ID starting with 1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83 not found: ID does not exist" containerID="1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83"
Sep 09 21:53:56.395199 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.395124    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83"} err="failed to get container status \"1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83\": rpc error: code = NotFound desc = could not find container \"1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83\": container with ID starting with 1cfd5f1ce661acbbd6f1482ef95ac8a2011cf52584b143c430fc74cb93ffec83 not found: ID does not exist"
Sep 09 21:53:56.397214 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.397182    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"]
Sep 09 21:53:56.401331 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.401307    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da0a18-5bf6777478-g5mqw"]
Sep 09 21:53:56.412115 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.412088    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"]
Sep 09 21:53:56.415949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.415886    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da0a18-6fc78c9597-27mdl"]
Sep 09 21:53:56.480904 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.480865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-dshm\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.481074 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.480938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-model-cache\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.481074 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.480987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-home\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.481154 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.481077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f639f-c697-49bb-9254-faeff75df20a-tls-certs\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.481154 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.481121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hmf7\" (UniqueName: \"kubernetes.io/projected/ac6f639f-c697-49bb-9254-faeff75df20a-kube-api-access-2hmf7\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.481154 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.481140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-tmp-dir\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.581864 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.581827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-model-cache\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.581864 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.581869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-home\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.582094 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.581910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f639f-c697-49bb-9254-faeff75df20a-tls-certs\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.582094 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.581933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hmf7\" (UniqueName: \"kubernetes.io/projected/ac6f639f-c697-49bb-9254-faeff75df20a-kube-api-access-2hmf7\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.582094 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.581966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-tmp-dir\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.582094 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.582039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-dshm\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.582279 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.582252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-model-cache\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.582345 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.582289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-home\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.582392 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.582340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-tmp-dir\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.584348 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.584324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-dshm\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.584454 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.584417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f639f-c697-49bb-9254-faeff75df20a-tls-certs\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.590201 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.590177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hmf7\" (UniqueName: \"kubernetes.io/projected/ac6f639f-c697-49bb-9254-faeff75df20a-kube-api-access-2hmf7\") pod \"noten-model-749a53-kserve-d5bdcd88c-bjhtj\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") " pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.695425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.695340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:53:56.966335 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:56.966308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"]
Sep 09 21:53:56.968573 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:53:56.968537    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac6f639f_c697_49bb_9254_faeff75df20a.slice/crio-6e836f555e8816ec14ccf223117dc15605a7d4d01dc3782787f13d9b1ddbad3a WatchSource:0}: Error finding container 6e836f555e8816ec14ccf223117dc15605a7d4d01dc3782787f13d9b1ddbad3a: Status 404 returned error can't find the container with id 6e836f555e8816ec14ccf223117dc15605a7d4d01dc3782787f13d9b1ddbad3a
Sep 09 21:53:57.047056 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:57.047026    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33afc968-7590-4088-b21b-73b69e6008fc" path="/var/lib/kubelet/pods/33afc968-7590-4088-b21b-73b69e6008fc/volumes"
Sep 09 21:53:57.047404 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:57.047391    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afc84ea3-4140-4e93-8dbe-01eec364d55f" path="/var/lib/kubelet/pods/afc84ea3-4140-4e93-8dbe-01eec364d55f/volumes"
Sep 09 21:53:57.047694 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:57.047682    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d64ef116-7daa-430f-b4c4-e83a10369c02" path="/var/lib/kubelet/pods/d64ef116-7daa-430f-b4c4-e83a10369c02/volumes"
Sep 09 21:53:57.381923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:57.381886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj" event={"ID":"ac6f639f-c697-49bb-9254-faeff75df20a","Type":"ContainerStarted","Data":"396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2"}
Sep 09 21:53:57.381923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:57.381926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj" event={"ID":"ac6f639f-c697-49bb-9254-faeff75df20a","Type":"ContainerStarted","Data":"6e836f555e8816ec14ccf223117dc15605a7d4d01dc3782787f13d9b1ddbad3a"}
Sep 09 21:53:57.398988 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:53:57.398941    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj" podStartSLOduration=1.398926681 podStartE2EDuration="1.398926681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:53:57.397116129 +0000 UTC m=+1242.882999169" watchObservedRunningTime="2026-09-09 21:53:57.398926681 +0000 UTC m=+1242.884809720"
Sep 09 21:54:01.165863 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:01.165498    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"]
Sep 09 21:54:06.696159 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:06.696067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:54:06.696159 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:06.696122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:54:06.708475 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:06.708447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:54:07.447246 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:07.447208    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:54:15.148222 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:15.148187    2563 scope.go:117] "RemoveContainer" containerID="5a1205b80d4192d52d63bf9a1cdd0507d3da48e06761c6bb01a26de635e0067a"
Sep 09 21:54:16.190286 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.190239    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf" podUID="f20aae99-46b9-481e-bd39-b5ade73cf856" containerName="main" containerID="cri-o://30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694" gracePeriod=45
Sep 09 21:54:16.432884 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.432853    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:54:16.484642 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.484547    2563 generic.go:356] "Generic (PLEG): container finished" podID="f20aae99-46b9-481e-bd39-b5ade73cf856" containerID="30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694" exitCode=0
Sep 09 21:54:16.484642 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.484589    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"
Sep 09 21:54:16.484848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.484635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf" event={"ID":"f20aae99-46b9-481e-bd39-b5ade73cf856","Type":"ContainerDied","Data":"30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694"}
Sep 09 21:54:16.484848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.484677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf" event={"ID":"f20aae99-46b9-481e-bd39-b5ade73cf856","Type":"ContainerDied","Data":"23296a57f810509be27907bddfcc6125bab5968116449a14d1636270c2de3c4a"}
Sep 09 21:54:16.484848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.484697    2563 scope.go:117] "RemoveContainer" containerID="30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694"
Sep 09 21:54:16.493665 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.493649    2563 scope.go:117] "RemoveContainer" containerID="30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694"
Sep 09 21:54:16.493958 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:54:16.493935    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694\": container with ID starting with 30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694 not found: ID does not exist" containerID="30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694"
Sep 09 21:54:16.494032 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.493972    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694"} err="failed to get container status \"30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694\": rpc error: code = NotFound desc = could not find container \"30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694\": container with ID starting with 30112c4bd9f9bee4324de8b406f7efb4aa1a970ecaee59fd48197a1d16874694 not found: ID does not exist"
Sep 09 21:54:16.586902 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.586868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20aae99-46b9-481e-bd39-b5ade73cf856-tls-certs\") pod \"f20aae99-46b9-481e-bd39-b5ade73cf856\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") "
Sep 09 21:54:16.586902 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.586910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-dshm\") pod \"f20aae99-46b9-481e-bd39-b5ade73cf856\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") "
Sep 09 21:54:16.587111 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.586949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-tmp-dir\") pod \"f20aae99-46b9-481e-bd39-b5ade73cf856\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") "
Sep 09 21:54:16.587111 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.586973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-model-cache\") pod \"f20aae99-46b9-481e-bd39-b5ade73cf856\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") "
Sep 09 21:54:16.587111 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.587035    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqtnm\" (UniqueName: \"kubernetes.io/projected/f20aae99-46b9-481e-bd39-b5ade73cf856-kube-api-access-zqtnm\") pod \"f20aae99-46b9-481e-bd39-b5ade73cf856\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") "
Sep 09 21:54:16.587263 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.587132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-home\") pod \"f20aae99-46b9-481e-bd39-b5ade73cf856\" (UID: \"f20aae99-46b9-481e-bd39-b5ade73cf856\") "
Sep 09 21:54:16.587326 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.587256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f20aae99-46b9-481e-bd39-b5ade73cf856" (UID: "f20aae99-46b9-481e-bd39-b5ade73cf856"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:54:16.587326 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.587269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-model-cache" (OuterVolumeSpecName: "model-cache") pod "f20aae99-46b9-481e-bd39-b5ade73cf856" (UID: "f20aae99-46b9-481e-bd39-b5ade73cf856"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:54:16.587434 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.587386    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-home" (OuterVolumeSpecName: "home") pod "f20aae99-46b9-481e-bd39-b5ade73cf856" (UID: "f20aae99-46b9-481e-bd39-b5ade73cf856"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:54:16.587572 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.587553    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:16.587620 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.587577    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:16.587620 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.587593    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:16.589377 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.589343    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20aae99-46b9-481e-bd39-b5ade73cf856-kube-api-access-zqtnm" (OuterVolumeSpecName: "kube-api-access-zqtnm") pod "f20aae99-46b9-481e-bd39-b5ade73cf856" (UID: "f20aae99-46b9-481e-bd39-b5ade73cf856"). InnerVolumeSpecName "kube-api-access-zqtnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:54:16.589377 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.589350    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-dshm" (OuterVolumeSpecName: "dshm") pod "f20aae99-46b9-481e-bd39-b5ade73cf856" (UID: "f20aae99-46b9-481e-bd39-b5ade73cf856"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:54:16.589572 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.589397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20aae99-46b9-481e-bd39-b5ade73cf856-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f20aae99-46b9-481e-bd39-b5ade73cf856" (UID: "f20aae99-46b9-481e-bd39-b5ade73cf856"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:54:16.689200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.689158    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtnm\" (UniqueName: \"kubernetes.io/projected/f20aae99-46b9-481e-bd39-b5ade73cf856-kube-api-access-zqtnm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:16.689200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.689192    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20aae99-46b9-481e-bd39-b5ade73cf856-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:16.689200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.689202    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20aae99-46b9-481e-bd39-b5ade73cf856-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:16.808606 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.808571    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"]
Sep 09 21:54:16.811546 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:16.811508    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-da0a18/test-model-da0a18-kserve-69cc89cbf9-fkmgf"]
Sep 09 21:54:17.047018 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:17.046937    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20aae99-46b9-481e-bd39-b5ade73cf856" path="/var/lib/kubelet/pods/f20aae99-46b9-481e-bd39-b5ade73cf856/volumes"
Sep 09 21:54:27.484852 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.484816    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"]
Sep 09 21:54:27.485443 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.485422    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20aae99-46b9-481e-bd39-b5ade73cf856" containerName="main"
Sep 09 21:54:27.485524 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.485449    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20aae99-46b9-481e-bd39-b5ade73cf856" containerName="main"
Sep 09 21:54:27.485578 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.485554    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20aae99-46b9-481e-bd39-b5ade73cf856" containerName="main"
Sep 09 21:54:27.490224 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.490203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.492957 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.492937    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-390abc\"/\"distinct-2-w0-390abc-kserve-self-signed-certs\""
Sep 09 21:54:27.497579 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.497558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"]
Sep 09 21:54:27.595706 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.595668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-dshm\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.595896 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.595746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tmp-dir\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.595896 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.595793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-home\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.595896 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.595845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-model-cache\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.596002 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.595910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crjl\" (UniqueName: \"kubernetes.io/projected/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-kube-api-access-2crjl\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.596002 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.595939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tls-certs\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.696502 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.696458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-model-cache\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.696701 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.696554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crjl\" (UniqueName: \"kubernetes.io/projected/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-kube-api-access-2crjl\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.696701 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.696589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tls-certs\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.696701 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.696643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-dshm\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.696701 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.696688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tmp-dir\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.696963 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.696778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-home\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.696963 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.696940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-model-cache\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.697093 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.697072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tmp-dir\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.697191 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.697173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-home\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.699038 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.699019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-dshm\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.699345 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.699330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tls-certs\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.704953 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.704929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crjl\" (UniqueName: \"kubernetes.io/projected/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-kube-api-access-2crjl\") pod \"distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") " pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:27.801565 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:27.801466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:28.075148 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:28.075120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"]
Sep 09 21:54:28.076551 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:54:28.076519    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a677d3a_7b7b_4c6d_b536_6a455103e7d7.slice/crio-f4db5ac855e3c40b00a0c3a9c17d06da81e82b40878e3bd4982cd821dbfa2761 WatchSource:0}: Error finding container f4db5ac855e3c40b00a0c3a9c17d06da81e82b40878e3bd4982cd821dbfa2761: Status 404 returned error can't find the container with id f4db5ac855e3c40b00a0c3a9c17d06da81e82b40878e3bd4982cd821dbfa2761
Sep 09 21:54:28.549689 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:28.549654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7" event={"ID":"0a677d3a-7b7b-4c6d-b536-6a455103e7d7","Type":"ContainerStarted","Data":"6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3"}
Sep 09 21:54:28.549689 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:28.549695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7" event={"ID":"0a677d3a-7b7b-4c6d-b536-6a455103e7d7","Type":"ContainerStarted","Data":"f4db5ac855e3c40b00a0c3a9c17d06da81e82b40878e3bd4982cd821dbfa2761"}
Sep 09 21:54:28.569415 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:28.569361    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7" podStartSLOduration=1.569343893 podStartE2EDuration="1.569343893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:28.567500239 +0000 UTC m=+1274.053383278" watchObservedRunningTime="2026-09-09 21:54:28.569343893 +0000 UTC m=+1274.055226930"
Sep 09 21:54:37.701124 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:37.701089    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"]
Sep 09 21:54:37.802372 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:37.802339    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:37.802372 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:37.802381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:37.815046 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:37.815021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:38.613678 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:38.613647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:54:44.465611 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.465572    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"]
Sep 09 21:54:44.469012 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.468991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:44.471768 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.471746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2d3bc8\""
Sep 09 21:54:44.480476 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.480452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"]
Sep 09 21:54:44.496650 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.496615    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"]
Sep 09 21:54:44.500157 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.500135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:44.503236 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.503216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2d3bc8-dockercfg-hh67r\""
Sep 09 21:54:44.514099 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.514074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"]
Sep 09 21:54:44.574353 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.574305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb\" (UID: \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:44.574353 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.574358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5rf\" (UniqueName: \"kubernetes.io/projected/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-kube-api-access-bx5rf\") pod \"payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb\" (UID: \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:44.574629 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.574416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb9db\" (UniqueName: \"kubernetes.io/projected/64335646-4cbc-4eaf-a4f2-3687fdea34fb-kube-api-access-jb9db\") pod \"payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6\" (UID: \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:44.574629 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.574533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64335646-4cbc-4eaf-a4f2-3687fdea34fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6\" (UID: \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:44.675580 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.675536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64335646-4cbc-4eaf-a4f2-3687fdea34fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6\" (UID: \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:44.675824 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.675598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb\" (UID: \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:44.675824 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.675626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5rf\" (UniqueName: \"kubernetes.io/projected/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-kube-api-access-bx5rf\") pod \"payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb\" (UID: \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:44.675824 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.675660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb9db\" (UniqueName: \"kubernetes.io/projected/64335646-4cbc-4eaf-a4f2-3687fdea34fb-kube-api-access-jb9db\") pod \"payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6\" (UID: \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:44.676332 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.676307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb\" (UID: \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:44.676420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.676334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64335646-4cbc-4eaf-a4f2-3687fdea34fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6\" (UID: \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:44.692659 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.692631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb9db\" (UniqueName: \"kubernetes.io/projected/64335646-4cbc-4eaf-a4f2-3687fdea34fb-kube-api-access-jb9db\") pod \"payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6\" (UID: \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:44.694942 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.694917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5rf\" (UniqueName: \"kubernetes.io/projected/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-kube-api-access-bx5rf\") pod \"payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb\" (UID: \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:44.781493 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.781411    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:44.811343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:44.811314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:45.064300 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.064277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"]
Sep 09 21:54:45.066174 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:54:45.066151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64335646_4cbc_4eaf_a4f2_3687fdea34fb.slice/crio-52be59b791dd4ce79bd713fbeefdb1ac06c9f977bf8edb8a3355d5b005433386 WatchSource:0}: Error finding container 52be59b791dd4ce79bd713fbeefdb1ac06c9f977bf8edb8a3355d5b005433386: Status 404 returned error can't find the container with id 52be59b791dd4ce79bd713fbeefdb1ac06c9f977bf8edb8a3355d5b005433386
Sep 09 21:54:45.099002 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.098975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"]
Sep 09 21:54:45.644037 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.644003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6" event={"ID":"64335646-4cbc-4eaf-a4f2-3687fdea34fb","Type":"ContainerStarted","Data":"64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e"}
Sep 09 21:54:45.644537 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.644052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6" event={"ID":"64335646-4cbc-4eaf-a4f2-3687fdea34fb","Type":"ContainerStarted","Data":"52be59b791dd4ce79bd713fbeefdb1ac06c9f977bf8edb8a3355d5b005433386"}
Sep 09 21:54:45.644537 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.644269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:54:45.645899 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.645860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb" event={"ID":"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961","Type":"ContainerStarted","Data":"03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356"}
Sep 09 21:54:45.646058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.645905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb" event={"ID":"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961","Type":"ContainerStarted","Data":"8f6e579ccd9f1396e72864d5f6e53381cb3ceba9e3fcf8e8eacf311f530cf118"}
Sep 09 21:54:45.646058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.646014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:45.663053 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.662999    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6" podStartSLOduration=1.6629785049999999 podStartE2EDuration="1.662978505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:45.661829762 +0000 UTC m=+1291.147712800" watchObservedRunningTime="2026-09-09 21:54:45.662978505 +0000 UTC m=+1291.148861546"
Sep 09 21:54:45.680258 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:45.680207    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb" podStartSLOduration=1.680190777 podStartE2EDuration="1.680190777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:45.679449619 +0000 UTC m=+1291.165332663" watchObservedRunningTime="2026-09-09 21:54:45.680190777 +0000 UTC m=+1291.166073809"
Sep 09 21:54:52.721384 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:52.721339    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj" podUID="ac6f639f-c697-49bb-9254-faeff75df20a" containerName="main" containerID="cri-o://396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2" gracePeriod=45
Sep 09 21:54:52.968497 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:52.968470    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:54:53.159835 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.159800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-model-cache\") pod \"ac6f639f-c697-49bb-9254-faeff75df20a\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") "
Sep 09 21:54:53.159835 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.159839    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f639f-c697-49bb-9254-faeff75df20a-tls-certs\") pod \"ac6f639f-c697-49bb-9254-faeff75df20a\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") "
Sep 09 21:54:53.160099 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.159874    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-dshm\") pod \"ac6f639f-c697-49bb-9254-faeff75df20a\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") "
Sep 09 21:54:53.160099 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.159903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-home\") pod \"ac6f639f-c697-49bb-9254-faeff75df20a\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") "
Sep 09 21:54:53.160099 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.159975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-tmp-dir\") pod \"ac6f639f-c697-49bb-9254-faeff75df20a\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") "
Sep 09 21:54:53.160099 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.160040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hmf7\" (UniqueName: \"kubernetes.io/projected/ac6f639f-c697-49bb-9254-faeff75df20a-kube-api-access-2hmf7\") pod \"ac6f639f-c697-49bb-9254-faeff75df20a\" (UID: \"ac6f639f-c697-49bb-9254-faeff75df20a\") "
Sep 09 21:54:53.160303 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.160142    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac6f639f-c697-49bb-9254-faeff75df20a" (UID: "ac6f639f-c697-49bb-9254-faeff75df20a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:54:53.160303 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.160224    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-home" (OuterVolumeSpecName: "home") pod "ac6f639f-c697-49bb-9254-faeff75df20a" (UID: "ac6f639f-c697-49bb-9254-faeff75df20a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:54:53.160303 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.160263    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac6f639f-c697-49bb-9254-faeff75df20a" (UID: "ac6f639f-c697-49bb-9254-faeff75df20a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:54:53.160470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.160452    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:53.160508 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.160477    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:53.160508 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.160491    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:53.162292 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.162267    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-dshm" (OuterVolumeSpecName: "dshm") pod "ac6f639f-c697-49bb-9254-faeff75df20a" (UID: "ac6f639f-c697-49bb-9254-faeff75df20a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:54:53.162381 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.162312    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6f639f-c697-49bb-9254-faeff75df20a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac6f639f-c697-49bb-9254-faeff75df20a" (UID: "ac6f639f-c697-49bb-9254-faeff75df20a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:54:53.162431 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.162391    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6f639f-c697-49bb-9254-faeff75df20a-kube-api-access-2hmf7" (OuterVolumeSpecName: "kube-api-access-2hmf7") pod "ac6f639f-c697-49bb-9254-faeff75df20a" (UID: "ac6f639f-c697-49bb-9254-faeff75df20a"). InnerVolumeSpecName "kube-api-access-2hmf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:54:53.261977 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.261941    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f639f-c697-49bb-9254-faeff75df20a-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:53.261977 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.261969    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hmf7\" (UniqueName: \"kubernetes.io/projected/ac6f639f-c697-49bb-9254-faeff75df20a-kube-api-access-2hmf7\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:53.261977 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.261980    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f639f-c697-49bb-9254-faeff75df20a-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:54:53.691301 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.691267    2563 generic.go:356] "Generic (PLEG): container finished" podID="ac6f639f-c697-49bb-9254-faeff75df20a" containerID="396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2" exitCode=0
Sep 09 21:54:53.691471 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.691312    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"
Sep 09 21:54:53.691471 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.691358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj" event={"ID":"ac6f639f-c697-49bb-9254-faeff75df20a","Type":"ContainerDied","Data":"396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2"}
Sep 09 21:54:53.691471 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.691406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj" event={"ID":"ac6f639f-c697-49bb-9254-faeff75df20a","Type":"ContainerDied","Data":"6e836f555e8816ec14ccf223117dc15605a7d4d01dc3782787f13d9b1ddbad3a"}
Sep 09 21:54:53.691471 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.691428    2563 scope.go:117] "RemoveContainer" containerID="396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2"
Sep 09 21:54:53.701574 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.701557    2563 scope.go:117] "RemoveContainer" containerID="396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2"
Sep 09 21:54:53.701852 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:54:53.701835    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2\": container with ID starting with 396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2 not found: ID does not exist" containerID="396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2"
Sep 09 21:54:53.701910 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.701859    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2"} err="failed to get container status \"396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2\": rpc error: code = NotFound desc = could not find container \"396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2\": container with ID starting with 396af894dff7650a10cfbfbc2112600b9fbb9ee5b070fc1ea29a9510f6345ee2 not found: ID does not exist"
Sep 09 21:54:53.713482 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.713454    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"]
Sep 09 21:54:53.719369 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:53.719349    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-749a53/noten-model-749a53-kserve-d5bdcd88c-bjhtj"]
Sep 09 21:54:55.046988 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:55.046953    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac6f639f-c697-49bb-9254-faeff75df20a" path="/var/lib/kubelet/pods/ac6f639f-c697-49bb-9254-faeff75df20a/volumes"
Sep 09 21:54:56.654375 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:56.654341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 21:54:56.654845 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:54:56.654424    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 21:55:11.305517 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.305480    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"]
Sep 09 21:55:11.306071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.305909    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6f639f-c697-49bb-9254-faeff75df20a" containerName="main"
Sep 09 21:55:11.306071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.305920    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6f639f-c697-49bb-9254-faeff75df20a" containerName="main"
Sep 09 21:55:11.306071 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.305999    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6f639f-c697-49bb-9254-faeff75df20a" containerName="main"
Sep 09 21:55:11.310483 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.310464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"
Sep 09 21:55:11.313397 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.313362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:55:11.317464 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.317439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"]
Sep 09 21:55:11.425884 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.425847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7381ffcf-9179-4513-9318-fe0579f7dbe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4883373b-m2bll\" (UID: \"7381ffcf-9179-4513-9318-fe0579f7dbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"
Sep 09 21:55:11.526679 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.526634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7381ffcf-9179-4513-9318-fe0579f7dbe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4883373b-m2bll\" (UID: \"7381ffcf-9179-4513-9318-fe0579f7dbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"
Sep 09 21:55:11.527339 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.527318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7381ffcf-9179-4513-9318-fe0579f7dbe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4883373b-m2bll\" (UID: \"7381ffcf-9179-4513-9318-fe0579f7dbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"
Sep 09 21:55:11.623573 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.623533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"
Sep 09 21:55:11.894842 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:11.894816    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"]
Sep 09 21:55:11.896770 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:55:11.896743    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7381ffcf_9179_4513_9318_fe0579f7dbe8.slice/crio-ae3b4d19a85c78efd33cd22c34e3eb1ab0c47c4e8f2664bf7deef2d230487e13 WatchSource:0}: Error finding container ae3b4d19a85c78efd33cd22c34e3eb1ab0c47c4e8f2664bf7deef2d230487e13: Status 404 returned error can't find the container with id ae3b4d19a85c78efd33cd22c34e3eb1ab0c47c4e8f2664bf7deef2d230487e13
Sep 09 21:55:12.803813 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.803772    2563 generic.go:356] "Generic (PLEG): container finished" podID="7381ffcf-9179-4513-9318-fe0579f7dbe8" containerID="6d19f0eb7f46a89e8c39779b11f9e206f58c185666cb89560da3dff03795c9a3" exitCode=0
Sep 09 21:55:12.804273 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.803816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll" event={"ID":"7381ffcf-9179-4513-9318-fe0579f7dbe8","Type":"ContainerDied","Data":"6d19f0eb7f46a89e8c39779b11f9e206f58c185666cb89560da3dff03795c9a3"}
Sep 09 21:55:12.804273 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.803894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll" event={"ID":"7381ffcf-9179-4513-9318-fe0579f7dbe8","Type":"ContainerStarted","Data":"ae3b4d19a85c78efd33cd22c34e3eb1ab0c47c4e8f2664bf7deef2d230487e13"}
Sep 09 21:55:12.907946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.907862    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"]
Sep 09 21:55:12.911587 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.911564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:12.914751 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.914702    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-64fec6-dockercfg-hxq97\""
Sep 09 21:55:12.914891 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.914773    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-64fec6\""
Sep 09 21:55:12.921766 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.921742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"]
Sep 09 21:55:12.940850 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.940818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5498223b-6218-4ff5-ae2b-aa4c95432465-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64\" (UID: \"5498223b-6218-4ff5-ae2b-aa4c95432465\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:12.941014 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:12.940943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfmn\" (UniqueName: \"kubernetes.io/projected/5498223b-6218-4ff5-ae2b-aa4c95432465-kube-api-access-9pfmn\") pod \"payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64\" (UID: \"5498223b-6218-4ff5-ae2b-aa4c95432465\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:13.041552 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.041515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfmn\" (UniqueName: \"kubernetes.io/projected/5498223b-6218-4ff5-ae2b-aa4c95432465-kube-api-access-9pfmn\") pod \"payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64\" (UID: \"5498223b-6218-4ff5-ae2b-aa4c95432465\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:13.041737 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.041599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5498223b-6218-4ff5-ae2b-aa4c95432465-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64\" (UID: \"5498223b-6218-4ff5-ae2b-aa4c95432465\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:13.042357 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.042337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5498223b-6218-4ff5-ae2b-aa4c95432465-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64\" (UID: \"5498223b-6218-4ff5-ae2b-aa4c95432465\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:13.050989 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.050961    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"]
Sep 09 21:55:13.053564 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.053536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfmn\" (UniqueName: \"kubernetes.io/projected/5498223b-6218-4ff5-ae2b-aa4c95432465-kube-api-access-9pfmn\") pod \"payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64\" (UID: \"5498223b-6218-4ff5-ae2b-aa4c95432465\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:13.054537 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.054517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.063310 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.063287    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"]
Sep 09 21:55:13.143224 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.143190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9z55\" (UniqueName: \"kubernetes.io/projected/549b94d9-1d2d-4b3b-9311-e961e8232c83-kube-api-access-j9z55\") pod \"payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94\" (UID: \"549b94d9-1d2d-4b3b-9311-e961e8232c83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.143386 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.143273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549b94d9-1d2d-4b3b-9311-e961e8232c83-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94\" (UID: \"549b94d9-1d2d-4b3b-9311-e961e8232c83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.223601 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.223503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:13.244823 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.244790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9z55\" (UniqueName: \"kubernetes.io/projected/549b94d9-1d2d-4b3b-9311-e961e8232c83-kube-api-access-j9z55\") pod \"payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94\" (UID: \"549b94d9-1d2d-4b3b-9311-e961e8232c83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.244963 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.244886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549b94d9-1d2d-4b3b-9311-e961e8232c83-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94\" (UID: \"549b94d9-1d2d-4b3b-9311-e961e8232c83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.245564 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.245542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549b94d9-1d2d-4b3b-9311-e961e8232c83-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94\" (UID: \"549b94d9-1d2d-4b3b-9311-e961e8232c83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.253259 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.253235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9z55\" (UniqueName: \"kubernetes.io/projected/549b94d9-1d2d-4b3b-9311-e961e8232c83-kube-api-access-j9z55\") pod \"payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94\" (UID: \"549b94d9-1d2d-4b3b-9311-e961e8232c83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.379886 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.379849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.501488 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.501457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"]
Sep 09 21:55:13.502860 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:55:13.502831    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5498223b_6218_4ff5_ae2b_aa4c95432465.slice/crio-09fbc741513393afc7b4eb7d8d1954ad3c094b53486a8440df040e9f0ead626f WatchSource:0}: Error finding container 09fbc741513393afc7b4eb7d8d1954ad3c094b53486a8440df040e9f0ead626f: Status 404 returned error can't find the container with id 09fbc741513393afc7b4eb7d8d1954ad3c094b53486a8440df040e9f0ead626f
Sep 09 21:55:13.658638 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.658612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"]
Sep 09 21:55:13.660669 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:55:13.660641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549b94d9_1d2d_4b3b_9311_e961e8232c83.slice/crio-0eda736f789144244aee5e22a8de8f9c55699c7e985c1a97961dfcb80d27b48c WatchSource:0}: Error finding container 0eda736f789144244aee5e22a8de8f9c55699c7e985c1a97961dfcb80d27b48c: Status 404 returned error can't find the container with id 0eda736f789144244aee5e22a8de8f9c55699c7e985c1a97961dfcb80d27b48c
Sep 09 21:55:13.812274 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.812175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94" event={"ID":"549b94d9-1d2d-4b3b-9311-e961e8232c83","Type":"ContainerStarted","Data":"61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8"}
Sep 09 21:55:13.812274 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.812228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94" event={"ID":"549b94d9-1d2d-4b3b-9311-e961e8232c83","Type":"ContainerStarted","Data":"0eda736f789144244aee5e22a8de8f9c55699c7e985c1a97961dfcb80d27b48c"}
Sep 09 21:55:13.812822 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.812305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:13.814140 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.814114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64" event={"ID":"5498223b-6218-4ff5-ae2b-aa4c95432465","Type":"ContainerStarted","Data":"04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0"}
Sep 09 21:55:13.814258 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.814164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64" event={"ID":"5498223b-6218-4ff5-ae2b-aa4c95432465","Type":"ContainerStarted","Data":"09fbc741513393afc7b4eb7d8d1954ad3c094b53486a8440df040e9f0ead626f"}
Sep 09 21:55:13.814258 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.814191    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:13.839601 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.839534    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94" podStartSLOduration=0.839512758 podStartE2EDuration="839.512758ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:13.835741209 +0000 UTC m=+1319.321624242" watchObservedRunningTime="2026-09-09 21:55:13.839512758 +0000 UTC m=+1319.325395797"
Sep 09 21:55:13.858590 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.858513    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64" podStartSLOduration=1.858497148 podStartE2EDuration="1.858497148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:13.85518105 +0000 UTC m=+1319.341064088" watchObservedRunningTime="2026-09-09 21:55:13.858497148 +0000 UTC m=+1319.344380186"
Sep 09 21:55:13.946515 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:13.946489    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"
Sep 09 21:55:14.051946 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:14.051907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7381ffcf-9179-4513-9318-fe0579f7dbe8-maas-api-service-ca\") pod \"7381ffcf-9179-4513-9318-fe0579f7dbe8\" (UID: \"7381ffcf-9179-4513-9318-fe0579f7dbe8\") "
Sep 09 21:55:14.052306 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:14.052279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7381ffcf-9179-4513-9318-fe0579f7dbe8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7381ffcf-9179-4513-9318-fe0579f7dbe8" (UID: "7381ffcf-9179-4513-9318-fe0579f7dbe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:14.153170 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:14.153133    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7381ffcf-9179-4513-9318-fe0579f7dbe8-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:55:14.823505 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:14.823474    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"
Sep 09 21:55:14.823997 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:14.823512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll" event={"ID":"7381ffcf-9179-4513-9318-fe0579f7dbe8","Type":"ContainerDied","Data":"ae3b4d19a85c78efd33cd22c34e3eb1ab0c47c4e8f2664bf7deef2d230487e13"}
Sep 09 21:55:14.823997 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:14.823568    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae3b4d19a85c78efd33cd22c34e3eb1ab0c47c4e8f2664bf7deef2d230487e13"
Sep 09 21:55:15.184131 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:15.184099    2563 scope.go:117] "RemoveContainer" containerID="6eda21e83de605546368955ee50c491982540f8c3dfa0310700f4fe5b1b345eb"
Sep 09 21:55:24.826035 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:24.826003    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 21:55:24.826486 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:24.826332    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 21:55:27.436390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:27.436356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"]
Sep 09 21:55:27.441519 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:27.441490    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4883373b-m2bll"]
Sep 09 21:55:29.046435 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:29.046401    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7381ffcf-9179-4513-9318-fe0579f7dbe8" path="/var/lib/kubelet/pods/7381ffcf-9179-4513-9318-fe0579f7dbe8/volumes"
Sep 09 21:55:35.417054 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:35.417013    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"]
Sep 09 21:55:50.438095 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.438054    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m" podUID="bd582094-39bc-40a7-a823-a7fd6da4229f" containerName="main" containerID="cri-o://b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2" gracePeriod=45
Sep 09 21:55:50.689203 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.689142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:55:50.719237 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719209    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-home\") pod \"bd582094-39bc-40a7-a823-a7fd6da4229f\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") "
Sep 09 21:55:50.719388 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-dshm\") pod \"bd582094-39bc-40a7-a823-a7fd6da4229f\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") "
Sep 09 21:55:50.719388 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719279    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582094-39bc-40a7-a823-a7fd6da4229f-tls-certs\") pod \"bd582094-39bc-40a7-a823-a7fd6da4229f\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") "
Sep 09 21:55:50.719388 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-tmp-dir\") pod \"bd582094-39bc-40a7-a823-a7fd6da4229f\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") "
Sep 09 21:55:50.719388 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719360    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-model-cache\") pod \"bd582094-39bc-40a7-a823-a7fd6da4229f\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") "
Sep 09 21:55:50.719630 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719403    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9h2s\" (UniqueName: \"kubernetes.io/projected/bd582094-39bc-40a7-a823-a7fd6da4229f-kube-api-access-v9h2s\") pod \"bd582094-39bc-40a7-a823-a7fd6da4229f\" (UID: \"bd582094-39bc-40a7-a823-a7fd6da4229f\") "
Sep 09 21:55:50.719630 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-home" (OuterVolumeSpecName: "home") pod "bd582094-39bc-40a7-a823-a7fd6da4229f" (UID: "bd582094-39bc-40a7-a823-a7fd6da4229f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:50.719779 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719672    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd582094-39bc-40a7-a823-a7fd6da4229f" (UID: "bd582094-39bc-40a7-a823-a7fd6da4229f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:50.719865 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719839    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd582094-39bc-40a7-a823-a7fd6da4229f" (UID: "bd582094-39bc-40a7-a823-a7fd6da4229f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:50.720005 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719916    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:55:50.720005 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.719935    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:55:50.721802 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.721773    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-dshm" (OuterVolumeSpecName: "dshm") pod "bd582094-39bc-40a7-a823-a7fd6da4229f" (UID: "bd582094-39bc-40a7-a823-a7fd6da4229f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:50.721907 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.721852    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd582094-39bc-40a7-a823-a7fd6da4229f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd582094-39bc-40a7-a823-a7fd6da4229f" (UID: "bd582094-39bc-40a7-a823-a7fd6da4229f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:55:50.722065 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.722042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd582094-39bc-40a7-a823-a7fd6da4229f-kube-api-access-v9h2s" (OuterVolumeSpecName: "kube-api-access-v9h2s") pod "bd582094-39bc-40a7-a823-a7fd6da4229f" (UID: "bd582094-39bc-40a7-a823-a7fd6da4229f"). InnerVolumeSpecName "kube-api-access-v9h2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:50.820694 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.820659    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:55:50.820694 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.820688    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582094-39bc-40a7-a823-a7fd6da4229f-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:55:50.820694 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.820699    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582094-39bc-40a7-a823-a7fd6da4229f-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:55:50.820935 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:50.820709    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9h2s\" (UniqueName: \"kubernetes.io/projected/bd582094-39bc-40a7-a823-a7fd6da4229f-kube-api-access-v9h2s\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:55:51.029556 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.029462    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd582094-39bc-40a7-a823-a7fd6da4229f" containerID="b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2" exitCode=0
Sep 09 21:55:51.029556 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.029510    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"
Sep 09 21:55:51.029556 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.029510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m" event={"ID":"bd582094-39bc-40a7-a823-a7fd6da4229f","Type":"ContainerDied","Data":"b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2"}
Sep 09 21:55:51.029556 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.029552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m" event={"ID":"bd582094-39bc-40a7-a823-a7fd6da4229f","Type":"ContainerDied","Data":"cac880ca174671f4e511c93a2421faa4dce69738ec6cc38fb4abcb90fe712c92"}
Sep 09 21:55:51.029897 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.029573    2563 scope.go:117] "RemoveContainer" containerID="b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2"
Sep 09 21:55:51.039542 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.039512    2563 scope.go:117] "RemoveContainer" containerID="b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2"
Sep 09 21:55:51.039861 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:55:51.039838    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2\": container with ID starting with b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2 not found: ID does not exist" containerID="b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2"
Sep 09 21:55:51.039939 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.039870    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2"} err="failed to get container status \"b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2\": rpc error: code = NotFound desc = could not find container \"b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2\": container with ID starting with b1b8fe89344680d64408b40bda261761a3e0c0803cccaad12c28fda7f73793c2 not found: ID does not exist"
Sep 09 21:55:51.050516 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.050491    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"]
Sep 09 21:55:51.054429 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:51.054406    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4883373b/e2e-lifecycle-model-4883373b-kserve-5b484c77f5-nfl5m"]
Sep 09 21:55:53.046481 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:55:53.046444    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd582094-39bc-40a7-a823-a7fd6da4229f" path="/var/lib/kubelet/pods/bd582094-39bc-40a7-a823-a7fd6da4229f/volumes"
Sep 09 21:56:05.996337 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:05.996296    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"]
Sep 09 21:56:05.997013 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:05.996993    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd582094-39bc-40a7-a823-a7fd6da4229f" containerName="main"
Sep 09 21:56:05.997068 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:05.997018    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd582094-39bc-40a7-a823-a7fd6da4229f" containerName="main"
Sep 09 21:56:05.997068 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:05.997065    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7381ffcf-9179-4513-9318-fe0579f7dbe8" containerName="revoke-keys"
Sep 09 21:56:05.997142 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:05.997074    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7381ffcf-9179-4513-9318-fe0579f7dbe8" containerName="revoke-keys"
Sep 09 21:56:05.997188 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:05.997176    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd582094-39bc-40a7-a823-a7fd6da4229f" containerName="main"
Sep 09 21:56:05.997224 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:05.997193    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7381ffcf-9179-4513-9318-fe0579f7dbe8" containerName="revoke-keys"
Sep 09 21:56:06.002204 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.002180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.005115 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.005092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-390abc\"/\"unconfigured-w0-390abc-kserve-self-signed-certs\""
Sep 09 21:56:06.010004 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.009978    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"]
Sep 09 21:56:06.077907 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.077867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-model-cache\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.078085 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.077925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tls-certs\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.078085 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.077970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tmp-dir\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.078085 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.078019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-dshm\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.078085 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.078048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-home\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.078085 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.078070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fznw\" (UniqueName: \"kubernetes.io/projected/b203b3b0-cc3c-4101-9570-16f7c6caed1d-kube-api-access-5fznw\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.179425 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.179389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-dshm\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.179634 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.179441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-home\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.179634 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.179485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fznw\" (UniqueName: \"kubernetes.io/projected/b203b3b0-cc3c-4101-9570-16f7c6caed1d-kube-api-access-5fznw\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.179796 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.179768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-model-cache\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.179923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.179901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-home\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.180053 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.180029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tls-certs\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.180121 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.180050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-model-cache\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.180181 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.180122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tmp-dir\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.180457 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.180431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tmp-dir\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.181915 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.181896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-dshm\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.182431 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.182410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tls-certs\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.190748 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.190702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fznw\" (UniqueName: \"kubernetes.io/projected/b203b3b0-cc3c-4101-9570-16f7c6caed1d-kube-api-access-5fznw\") pod \"unconfigured-w0-390abc-kserve-b7584874d-8nfrz\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") " pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.314074 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.313971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:06.589079 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:06.589051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"]
Sep 09 21:56:06.590560 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:56:06.590524    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb203b3b0_cc3c_4101_9570_16f7c6caed1d.slice/crio-c8329838d5ce2738c47541f7ea92ba154f42289432c3b4e5efbd74603a530d37 WatchSource:0}: Error finding container c8329838d5ce2738c47541f7ea92ba154f42289432c3b4e5efbd74603a530d37: Status 404 returned error can't find the container with id c8329838d5ce2738c47541f7ea92ba154f42289432c3b4e5efbd74603a530d37
Sep 09 21:56:07.115596 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:07.115561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz" event={"ID":"b203b3b0-cc3c-4101-9570-16f7c6caed1d","Type":"ContainerStarted","Data":"b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408"}
Sep 09 21:56:07.115983 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:07.115608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz" event={"ID":"b203b3b0-cc3c-4101-9570-16f7c6caed1d","Type":"ContainerStarted","Data":"c8329838d5ce2738c47541f7ea92ba154f42289432c3b4e5efbd74603a530d37"}
Sep 09 21:56:07.134175 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:07.134122    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz" podStartSLOduration=2.134107431 podStartE2EDuration="2.134107431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:07.130686904 +0000 UTC m=+1372.616569943" watchObservedRunningTime="2026-09-09 21:56:07.134107431 +0000 UTC m=+1372.619990469"
Sep 09 21:56:16.314497 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:16.314461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:16.314923 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:16.314533    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:16.327439 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:16.327418    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:17.181255 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:17.181228    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:56:30.501322 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.501284    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"]
Sep 09 21:56:30.504882 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.504864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.508819 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.508791    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"auth-test-model-2d3bc8-kserve-self-signed-certs\""
Sep 09 21:56:30.508976 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.508880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"kube-root-ca.crt\""
Sep 09 21:56:30.508976 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.508892    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"default-dockercfg-s9tcs\""
Sep 09 21:56:30.508976 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.508881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"openshift-service-ca.crt\""
Sep 09 21:56:30.514576 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.514555    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"]
Sep 09 21:56:30.526190 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.526164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-home\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.526343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.526196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-model-cache\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.526343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.526220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tmp-dir\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.526343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.526246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467fv\" (UniqueName: \"kubernetes.io/projected/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-kube-api-access-467fv\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.526343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.526294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tls-certs\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.526490 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.526388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-dshm\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.627550 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.627504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-dshm\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.627798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.627589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-home\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.627798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.627612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-model-cache\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.627798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.627640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tmp-dir\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.627798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.627662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467fv\" (UniqueName: \"kubernetes.io/projected/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-kube-api-access-467fv\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.627798 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.627747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tls-certs\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.628097 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.628072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-home\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.628160 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.628117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-model-cache\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.628160 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.628142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tmp-dir\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.629977 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.629955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-dshm\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.630464 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.630443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tls-certs\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.635852 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.635831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-467fv\" (UniqueName: \"kubernetes.io/projected/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-kube-api-access-467fv\") pod \"auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:30.815741 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:30.815627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:31.087089 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:31.087061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"]
Sep 09 21:56:31.089273 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:56:31.089243    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b284dfd_a5c5_4199_8222_f30ee1bd62b4.slice/crio-c7dd71bb4dc980673a57be8e2687ac6a1ebff72751e0bac2df78a6ba3e501d64 WatchSource:0}: Error finding container c7dd71bb4dc980673a57be8e2687ac6a1ebff72751e0bac2df78a6ba3e501d64: Status 404 returned error can't find the container with id c7dd71bb4dc980673a57be8e2687ac6a1ebff72751e0bac2df78a6ba3e501d64
Sep 09 21:56:31.249089 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:31.249049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4" event={"ID":"5b284dfd-a5c5-4199-8222-f30ee1bd62b4","Type":"ContainerStarted","Data":"ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718"}
Sep 09 21:56:31.249089 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:31.249098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4" event={"ID":"5b284dfd-a5c5-4199-8222-f30ee1bd62b4","Type":"ContainerStarted","Data":"c7dd71bb4dc980673a57be8e2687ac6a1ebff72751e0bac2df78a6ba3e501d64"}
Sep 09 21:56:31.269184 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:31.269139    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4" podStartSLOduration=1.269126237 podStartE2EDuration="1.269126237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:31.267048999 +0000 UTC m=+1396.752932038" watchObservedRunningTime="2026-09-09 21:56:31.269126237 +0000 UTC m=+1396.755009274"
Sep 09 21:56:40.816246 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:40.816201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:40.816910 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:40.816260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:40.829469 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:40.829444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:56:41.313904 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:56:41.313875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 21:57:15.227087 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:57:15.227003    2563 scope.go:117] "RemoveContainer" containerID="f1f83a955e546d16e89f79cc67ab0b55b0e2a6d0e7e662a362558e72de9c81a2"
Sep 09 21:58:15.248210 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:15.248175    2563 scope.go:117] "RemoveContainer" containerID="57f187660bbbfa26c964533f0e425f883e8a7b123e3984b169c6fd98da876bff"
Sep 09 21:58:21.288535 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.288496    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"]
Sep 09 21:58:21.293652 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.293632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:21.297135 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.297112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9dedad-dockercfg-jtsf4\""
Sep 09 21:58:21.297266 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.297178    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9dedad\""
Sep 09 21:58:21.313627 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.313597    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"]
Sep 09 21:58:21.386743 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.386682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdwc\" (UniqueName: \"kubernetes.io/projected/4f30fe77-08f9-4804-9f76-1258a5fd077c-kube-api-access-mgdwc\") pod \"payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8\" (UID: \"4f30fe77-08f9-4804-9f76-1258a5fd077c\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:21.386914 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.386771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f30fe77-08f9-4804-9f76-1258a5fd077c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8\" (UID: \"4f30fe77-08f9-4804-9f76-1258a5fd077c\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:21.488182 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.488145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdwc\" (UniqueName: \"kubernetes.io/projected/4f30fe77-08f9-4804-9f76-1258a5fd077c-kube-api-access-mgdwc\") pod \"payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8\" (UID: \"4f30fe77-08f9-4804-9f76-1258a5fd077c\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:21.488182 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.488183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f30fe77-08f9-4804-9f76-1258a5fd077c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8\" (UID: \"4f30fe77-08f9-4804-9f76-1258a5fd077c\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:21.488929 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.488910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f30fe77-08f9-4804-9f76-1258a5fd077c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8\" (UID: \"4f30fe77-08f9-4804-9f76-1258a5fd077c\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:21.496597 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.496577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdwc\" (UniqueName: \"kubernetes.io/projected/4f30fe77-08f9-4804-9f76-1258a5fd077c-kube-api-access-mgdwc\") pod \"payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8\" (UID: \"4f30fe77-08f9-4804-9f76-1258a5fd077c\") " pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:21.604529 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.604496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:21.879198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:21.879173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"]
Sep 09 21:58:21.880345 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:58:21.880320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f30fe77_08f9_4804_9f76_1258a5fd077c.slice/crio-7bd9259c672adbd2ec07c3ac6d6deb83a132df89dbdd85912c92b32b19f7693c WatchSource:0}: Error finding container 7bd9259c672adbd2ec07c3ac6d6deb83a132df89dbdd85912c92b32b19f7693c: Status 404 returned error can't find the container with id 7bd9259c672adbd2ec07c3ac6d6deb83a132df89dbdd85912c92b32b19f7693c
Sep 09 21:58:22.871814 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:22.871779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8" event={"ID":"4f30fe77-08f9-4804-9f76-1258a5fd077c","Type":"ContainerStarted","Data":"139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228"}
Sep 09 21:58:22.872248 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:22.871835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8" event={"ID":"4f30fe77-08f9-4804-9f76-1258a5fd077c","Type":"ContainerStarted","Data":"7bd9259c672adbd2ec07c3ac6d6deb83a132df89dbdd85912c92b32b19f7693c"}
Sep 09 21:58:22.872248 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:22.871937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:22.890371 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:22.890322    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8" podStartSLOduration=1.890306511 podStartE2EDuration="1.890306511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:58:22.888907401 +0000 UTC m=+1508.374790439" watchObservedRunningTime="2026-09-09 21:58:22.890306511 +0000 UTC m=+1508.376189550"
Sep 09 21:58:33.880195 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:33.880157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:58:43.672113 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:43.672076    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"]
Sep 09 21:58:43.675509 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:43.675491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"
Sep 09 21:58:43.678357 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:43.678337    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:58:43.686694 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:43.686673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"]
Sep 09 21:58:43.720219 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:43.720189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2807109-c471-4cf9-bed4-1dea25ae5208-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5\" (UID: \"f2807109-c471-4cf9-bed4-1dea25ae5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"
Sep 09 21:58:43.821648 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:43.821609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2807109-c471-4cf9-bed4-1dea25ae5208-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5\" (UID: \"f2807109-c471-4cf9-bed4-1dea25ae5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"
Sep 09 21:58:43.822276 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:43.822248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2807109-c471-4cf9-bed4-1dea25ae5208-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5\" (UID: \"f2807109-c471-4cf9-bed4-1dea25ae5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"
Sep 09 21:58:43.987644 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:43.987565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"
Sep 09 21:58:44.264094 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:44.264044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"]
Sep 09 21:58:44.266922 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:58:44.266890    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2807109_c471_4cf9_bed4_1dea25ae5208.slice/crio-885844dc39e222ee99734b3313c317c8a9e66a632a832ac93bb57a8ec339cfe6 WatchSource:0}: Error finding container 885844dc39e222ee99734b3313c317c8a9e66a632a832ac93bb57a8ec339cfe6: Status 404 returned error can't find the container with id 885844dc39e222ee99734b3313c317c8a9e66a632a832ac93bb57a8ec339cfe6
Sep 09 21:58:44.996782 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:44.996745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5" event={"ID":"f2807109-c471-4cf9-bed4-1dea25ae5208","Type":"ContainerStarted","Data":"fd8c708950950918b5364d7d924d927002802737d7df658528674e575bd43f2d"}
Sep 09 21:58:44.997206 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:44.996838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5" event={"ID":"f2807109-c471-4cf9-bed4-1dea25ae5208","Type":"ContainerStarted","Data":"885844dc39e222ee99734b3313c317c8a9e66a632a832ac93bb57a8ec339cfe6"}
Sep 09 21:58:45.022447 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:45.022386    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5" podStartSLOduration=2.022370853 podStartE2EDuration="2.022370853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:58:45.018930397 +0000 UTC m=+1530.504813428" watchObservedRunningTime="2026-09-09 21:58:45.022370853 +0000 UTC m=+1530.508253892"
Sep 09 21:58:46.004324 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:46.004288    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2807109-c471-4cf9-bed4-1dea25ae5208" containerID="fd8c708950950918b5364d7d924d927002802737d7df658528674e575bd43f2d" exitCode=0
Sep 09 21:58:46.004777 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:46.004374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5" event={"ID":"f2807109-c471-4cf9-bed4-1dea25ae5208","Type":"ContainerDied","Data":"fd8c708950950918b5364d7d924d927002802737d7df658528674e575bd43f2d"}
Sep 09 21:58:46.823132 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:46.823095    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"]
Sep 09 21:58:46.826864 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:46.826841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"
Sep 09 21:58:46.838273 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:46.838244    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"]
Sep 09 21:58:46.853820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:46.853783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef55f77-dfa1-49f4-a410-03efac73028d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d8401-84t4r\" (UID: \"4ef55f77-dfa1-49f4-a410-03efac73028d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"
Sep 09 21:58:46.955012 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:46.954971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef55f77-dfa1-49f4-a410-03efac73028d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d8401-84t4r\" (UID: \"4ef55f77-dfa1-49f4-a410-03efac73028d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"
Sep 09 21:58:46.955588 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:46.955567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef55f77-dfa1-49f4-a410-03efac73028d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d8401-84t4r\" (UID: \"4ef55f77-dfa1-49f4-a410-03efac73028d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"
Sep 09 21:58:47.138228 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:47.138196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"
Sep 09 21:58:47.150550 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:47.150526    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"
Sep 09 21:58:47.258463 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:47.258428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2807109-c471-4cf9-bed4-1dea25ae5208-maas-api-service-ca\") pod \"f2807109-c471-4cf9-bed4-1dea25ae5208\" (UID: \"f2807109-c471-4cf9-bed4-1dea25ae5208\") "
Sep 09 21:58:47.258801 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:47.258779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2807109-c471-4cf9-bed4-1dea25ae5208-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2807109-c471-4cf9-bed4-1dea25ae5208" (UID: "f2807109-c471-4cf9-bed4-1dea25ae5208"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:47.360081 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:47.360047    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2807109-c471-4cf9-bed4-1dea25ae5208-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:58:47.425596 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:58:47.425358    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef55f77_dfa1_49f4_a410_03efac73028d.slice/crio-ae041be189964cef41a06f40c6c9eb8676198cdbc32768c89ba62129cb371cab WatchSource:0}: Error finding container ae041be189964cef41a06f40c6c9eb8676198cdbc32768c89ba62129cb371cab: Status 404 returned error can't find the container with id ae041be189964cef41a06f40c6c9eb8676198cdbc32768c89ba62129cb371cab
Sep 09 21:58:47.425596 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:47.425380    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"]
Sep 09 21:58:48.017676 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.017571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r" event={"ID":"4ef55f77-dfa1-49f4-a410-03efac73028d","Type":"ContainerStarted","Data":"e474d81d178b279ad331739912166853438bd24f34663d1646594cf714512863"}
Sep 09 21:58:48.017676 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.017639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r" event={"ID":"4ef55f77-dfa1-49f4-a410-03efac73028d","Type":"ContainerStarted","Data":"ae041be189964cef41a06f40c6c9eb8676198cdbc32768c89ba62129cb371cab"}
Sep 09 21:58:48.019039 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.019019    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"
Sep 09 21:58:48.019145 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.019060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5" event={"ID":"f2807109-c471-4cf9-bed4-1dea25ae5208","Type":"ContainerDied","Data":"885844dc39e222ee99734b3313c317c8a9e66a632a832ac93bb57a8ec339cfe6"}
Sep 09 21:58:48.019145 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.019108    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885844dc39e222ee99734b3313c317c8a9e66a632a832ac93bb57a8ec339cfe6"
Sep 09 21:58:48.033687 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.033641    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r" podStartSLOduration=2.033624554 podStartE2EDuration="2.033624554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:58:48.032800974 +0000 UTC m=+1533.518684204" watchObservedRunningTime="2026-09-09 21:58:48.033624554 +0000 UTC m=+1533.519507591"
Sep 09 21:58:48.499916 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.499882    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"]
Sep 09 21:58:48.506364 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.506326    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"]
Sep 09 21:58:48.531468 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.531435    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"]
Sep 09 21:58:48.541343 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.541311    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"]
Sep 09 21:58:48.558674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:48.558639    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"]
Sep 09 21:58:49.029538 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:49.029496    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ef55f77-dfa1-49f4-a410-03efac73028d" containerID="e474d81d178b279ad331739912166853438bd24f34663d1646594cf714512863" exitCode=0
Sep 09 21:58:49.029785 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:49.029684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r" event={"ID":"4ef55f77-dfa1-49f4-a410-03efac73028d","Type":"ContainerDied","Data":"e474d81d178b279ad331739912166853438bd24f34663d1646594cf714512863"}
Sep 09 21:58:50.174671 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:50.174642    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"
Sep 09 21:58:50.293805 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:50.293683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef55f77-dfa1-49f4-a410-03efac73028d-maas-api-service-ca\") pod \"4ef55f77-dfa1-49f4-a410-03efac73028d\" (UID: \"4ef55f77-dfa1-49f4-a410-03efac73028d\") "
Sep 09 21:58:50.294058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:50.294034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef55f77-dfa1-49f4-a410-03efac73028d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ef55f77-dfa1-49f4-a410-03efac73028d" (UID: "4ef55f77-dfa1-49f4-a410-03efac73028d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:50.395416 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:50.395378    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef55f77-dfa1-49f4-a410-03efac73028d-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:58:51.041363 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:51.041330    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"
Sep 09 21:58:51.047823 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:51.047786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r" event={"ID":"4ef55f77-dfa1-49f4-a410-03efac73028d","Type":"ContainerDied","Data":"ae041be189964cef41a06f40c6c9eb8676198cdbc32768c89ba62129cb371cab"}
Sep 09 21:58:51.047964 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:51.047847    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae041be189964cef41a06f40c6c9eb8676198cdbc32768c89ba62129cb371cab"
Sep 09 21:58:55.394288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.394253    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"]
Sep 09 21:58:55.394786 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.394769    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2807109-c471-4cf9-bed4-1dea25ae5208" containerName="revoke-keys"
Sep 09 21:58:55.394858 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.394790    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2807109-c471-4cf9-bed4-1dea25ae5208" containerName="revoke-keys"
Sep 09 21:58:55.394858 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.394810    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef55f77-dfa1-49f4-a410-03efac73028d" containerName="revoke-keys"
Sep 09 21:58:55.394858 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.394815    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef55f77-dfa1-49f4-a410-03efac73028d" containerName="revoke-keys"
Sep 09 21:58:55.394974 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.394902    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef55f77-dfa1-49f4-a410-03efac73028d" containerName="revoke-keys"
Sep 09 21:58:55.394974 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.394914    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2807109-c471-4cf9-bed4-1dea25ae5208" containerName="revoke-keys"
Sep 09 21:58:55.398568 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.398540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.401599 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.401575    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"e2e-conflict-a-d5ffdfe0-kserve-self-signed-certs\""
Sep 09 21:58:55.407667 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.407644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"]
Sep 09 21:58:55.449022 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.448995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn289\" (UniqueName: \"kubernetes.io/projected/6cdf9dcc-148a-44df-a088-4c1836a57890-kube-api-access-sn289\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.449198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.449050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-model-cache\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.449198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.449072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdf9dcc-148a-44df-a088-4c1836a57890-tls-certs\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.449198 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.449171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-dshm\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.449345 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.449206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-home\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.449345 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.449222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-tmp-dir\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-model-cache\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550200 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdf9dcc-148a-44df-a088-4c1836a57890-tls-certs\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-dshm\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-home\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550442 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-tmp-dir\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550662 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-model-cache\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn289\" (UniqueName: \"kubernetes.io/projected/6cdf9dcc-148a-44df-a088-4c1836a57890-kube-api-access-sn289\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550767 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-home\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.550872 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.550757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-tmp-dir\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.552892 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.552857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-dshm\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.553138 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.553121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdf9dcc-148a-44df-a088-4c1836a57890-tls-certs\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.560252 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.560227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn289\" (UniqueName: \"kubernetes.io/projected/6cdf9dcc-148a-44df-a088-4c1836a57890-kube-api-access-sn289\") pod \"e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.711327 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.711238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:58:55.998035 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:55.998007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"]
Sep 09 21:58:56.000457 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:58:56.000432    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cdf9dcc_148a_44df_a088_4c1836a57890.slice/crio-b93581730d7a9b2bc8c72b147e5f1a709f34d55c0a08ba8af15e550129c0f031 WatchSource:0}: Error finding container b93581730d7a9b2bc8c72b147e5f1a709f34d55c0a08ba8af15e550129c0f031: Status 404 returned error can't find the container with id b93581730d7a9b2bc8c72b147e5f1a709f34d55c0a08ba8af15e550129c0f031
Sep 09 21:58:56.069450 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:56.069413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb" event={"ID":"6cdf9dcc-148a-44df-a088-4c1836a57890","Type":"ContainerStarted","Data":"9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b"}
Sep 09 21:58:56.069560 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:56.069465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb" event={"ID":"6cdf9dcc-148a-44df-a088-4c1836a57890","Type":"ContainerStarted","Data":"b93581730d7a9b2bc8c72b147e5f1a709f34d55c0a08ba8af15e550129c0f031"}
Sep 09 21:58:56.093845 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:56.093792    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb" podStartSLOduration=1.093775235 podStartE2EDuration="1.093775235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:58:56.090919443 +0000 UTC m=+1541.576802480" watchObservedRunningTime="2026-09-09 21:58:56.093775235 +0000 UTC m=+1541.579658317"
Sep 09 21:58:59.574936 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.574895    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"]
Sep 09 21:58:59.575399 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.575143    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4" podUID="6ff02b24-2e79-461a-a09c-1d267306d5a6" containerName="payload-processing" containerID="cri-o://965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93" gracePeriod=30
Sep 09 21:58:59.750656 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.750624    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"]
Sep 09 21:58:59.767694 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.761987    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-390abc-x5hz5"]
Sep 09 21:58:59.832018 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.831959    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:58:59.897266 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.897236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88bj5\" (UniqueName: \"kubernetes.io/projected/6ff02b24-2e79-461a-a09c-1d267306d5a6-kube-api-access-88bj5\") pod \"6ff02b24-2e79-461a-a09c-1d267306d5a6\" (UID: \"6ff02b24-2e79-461a-a09c-1d267306d5a6\") "
Sep 09 21:58:59.897452 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.897311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff02b24-2e79-461a-a09c-1d267306d5a6-plugins-config-volume\") pod \"6ff02b24-2e79-461a-a09c-1d267306d5a6\" (UID: \"6ff02b24-2e79-461a-a09c-1d267306d5a6\") "
Sep 09 21:58:59.897704 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.897680    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff02b24-2e79-461a-a09c-1d267306d5a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ff02b24-2e79-461a-a09c-1d267306d5a6" (UID: "6ff02b24-2e79-461a-a09c-1d267306d5a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:59.899533 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.899500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff02b24-2e79-461a-a09c-1d267306d5a6-kube-api-access-88bj5" (OuterVolumeSpecName: "kube-api-access-88bj5") pod "6ff02b24-2e79-461a-a09c-1d267306d5a6" (UID: "6ff02b24-2e79-461a-a09c-1d267306d5a6"). InnerVolumeSpecName "kube-api-access-88bj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:59.999039 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.999003    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88bj5\" (UniqueName: \"kubernetes.io/projected/6ff02b24-2e79-461a-a09c-1d267306d5a6-kube-api-access-88bj5\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:58:59.999039 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:58:59.999033    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff02b24-2e79-461a-a09c-1d267306d5a6-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:00.091831 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.091740    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ff02b24-2e79-461a-a09c-1d267306d5a6" containerID="965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93" exitCode=0
Sep 09 21:59:00.091831 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.091784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"
Sep 09 21:59:00.092003 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.091840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4" event={"ID":"6ff02b24-2e79-461a-a09c-1d267306d5a6","Type":"ContainerDied","Data":"965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93"}
Sep 09 21:59:00.092003 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.091904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4" event={"ID":"6ff02b24-2e79-461a-a09c-1d267306d5a6","Type":"ContainerDied","Data":"28bd8a188c857fa4b6ca84c21e516ebd2086298deb27d1a5e5ce566e3178b634"}
Sep 09 21:59:00.092003 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.091937    2563 scope.go:117] "RemoveContainer" containerID="965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93"
Sep 09 21:59:00.104203 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.104185    2563 scope.go:117] "RemoveContainer" containerID="965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93"
Sep 09 21:59:00.104466 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:59:00.104447    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93\": container with ID starting with 965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93 not found: ID does not exist" containerID="965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93"
Sep 09 21:59:00.104533 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.104475    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93"} err="failed to get container status \"965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93\": rpc error: code = NotFound desc = could not find container \"965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93\": container with ID starting with 965d9e3688c6181d25303412f3ac932d475e05622ac3003bbadf19ee2c82ef93 not found: ID does not exist"
Sep 09 21:59:00.118850 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.118818    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"]
Sep 09 21:59:00.122632 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:00.122608    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-390abc-59b465557f-nvtg4"]
Sep 09 21:59:01.046943 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:01.046909    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff02b24-2e79-461a-a09c-1d267306d5a6" path="/var/lib/kubelet/pods/6ff02b24-2e79-461a-a09c-1d267306d5a6/volumes"
Sep 09 21:59:01.047335 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:01.047238    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2807109-c471-4cf9-bed4-1dea25ae5208" path="/var/lib/kubelet/pods/f2807109-c471-4cf9-bed4-1dea25ae5208/volumes"
Sep 09 21:59:03.292503 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.292467    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"]
Sep 09 21:59:03.297471 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.297443    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8401-84t4r"]
Sep 09 21:59:03.521342 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.521299    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7" podUID="0a677d3a-7b7b-4c6d-b536-6a455103e7d7" containerName="main" containerID="cri-o://6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3" gracePeriod=45
Sep 09 21:59:03.526058 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.526022    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj" podUID="a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" containerName="main" containerID="cri-o://7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0" gracePeriod=45
Sep 09 21:59:03.551672 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.551635    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr" podUID="fde76298-df0e-49d0-927e-cd6d114e0320" containerName="main" containerID="cri-o://1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53" gracePeriod=45
Sep 09 21:59:03.561869 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.561829    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4" podUID="04e53c54-d895-4d11-800b-c0017ca20e3d" containerName="main" containerID="cri-o://d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6" gracePeriod=45
Sep 09 21:59:03.578167 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.578102    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz" podUID="b203b3b0-cc3c-4101-9570-16f7c6caed1d" containerName="main" containerID="cri-o://b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408" gracePeriod=45
Sep 09 21:59:03.841390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.841357    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:59:03.893615 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.893587    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:59:03.933273 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.933218    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:59:03.936826 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.936804    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:59:03.940368 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.940348    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:59:03.941223 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941207    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tmp-dir\") pod \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") "
Sep 09 21:59:03.941299 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-home\") pod \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") "
Sep 09 21:59:03.941299 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941295    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-model-cache\") pod \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") "
Sep 09 21:59:03.941420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tls-certs\") pod \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") "
Sep 09 21:59:03.941420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941340    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-dshm\") pod \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") "
Sep 09 21:59:03.941420 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-598dv\" (UniqueName: \"kubernetes.io/projected/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-kube-api-access-598dv\") pod \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\" (UID: \"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc\") "
Sep 09 21:59:03.941568 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941511    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" (UID: "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:03.941568 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941543    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-home" (OuterVolumeSpecName: "home") pod "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" (UID: "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:03.941821 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-model-cache" (OuterVolumeSpecName: "model-cache") pod "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" (UID: "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:03.941922 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941820    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:03.941922 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.941843    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:03.944041 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.944015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-dshm" (OuterVolumeSpecName: "dshm") pod "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" (UID: "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:03.944137 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.944106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" (UID: "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:03.944184 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:03.944154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-kube-api-access-598dv" (OuterVolumeSpecName: "kube-api-access-598dv") pod "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" (UID: "a940c2ee-0fc5-4d46-9d97-0689d9f10ecc"). InnerVolumeSpecName "kube-api-access-598dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:04.042306 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-home\") pod \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") "
Sep 09 21:59:04.042502 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-tmp-dir\") pod \"04e53c54-d895-4d11-800b-c0017ca20e3d\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") "
Sep 09 21:59:04.042502 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-home\") pod \"fde76298-df0e-49d0-927e-cd6d114e0320\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") "
Sep 09 21:59:04.042502 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e53c54-d895-4d11-800b-c0017ca20e3d-tls-certs\") pod \"04e53c54-d895-4d11-800b-c0017ca20e3d\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") "
Sep 09 21:59:04.042502 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp5lk\" (UniqueName: \"kubernetes.io/projected/04e53c54-d895-4d11-800b-c0017ca20e3d-kube-api-access-fp5lk\") pod \"04e53c54-d895-4d11-800b-c0017ca20e3d\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") "
Sep 09 21:59:04.042502 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlmnc\" (UniqueName: \"kubernetes.io/projected/fde76298-df0e-49d0-927e-cd6d114e0320-kube-api-access-dlmnc\") pod \"fde76298-df0e-49d0-927e-cd6d114e0320\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") "
Sep 09 21:59:04.042502 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042481    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-model-cache\") pod \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042515    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde76298-df0e-49d0-927e-cd6d114e0320-tls-certs\") pod \"fde76298-df0e-49d0-927e-cd6d114e0320\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2crjl\" (UniqueName: \"kubernetes.io/projected/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-kube-api-access-2crjl\") pod \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042579    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-dshm\") pod \"fde76298-df0e-49d0-927e-cd6d114e0320\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-home" (OuterVolumeSpecName: "home") pod "b203b3b0-cc3c-4101-9570-16f7c6caed1d" (UID: "b203b3b0-cc3c-4101-9570-16f7c6caed1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042568    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04e53c54-d895-4d11-800b-c0017ca20e3d" (UID: "04e53c54-d895-4d11-800b-c0017ca20e3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-home\") pod \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042643    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-home" (OuterVolumeSpecName: "home") pod "fde76298-df0e-49d0-927e-cd6d114e0320" (UID: "fde76298-df0e-49d0-927e-cd6d114e0320"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-model-cache\") pod \"04e53c54-d895-4d11-800b-c0017ca20e3d\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-home\") pod \"04e53c54-d895-4d11-800b-c0017ca20e3d\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fznw\" (UniqueName: \"kubernetes.io/projected/b203b3b0-cc3c-4101-9570-16f7c6caed1d-kube-api-access-5fznw\") pod \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tls-certs\") pod \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042852    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-model-cache\") pod \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") "
Sep 09 21:59:04.042870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-home" (OuterVolumeSpecName: "home") pod "0a677d3a-7b7b-4c6d-b536-6a455103e7d7" (UID: "0a677d3a-7b7b-4c6d-b536-6a455103e7d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042893    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b203b3b0-cc3c-4101-9570-16f7c6caed1d" (UID: "b203b3b0-cc3c-4101-9570-16f7c6caed1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-tmp-dir\") pod \"fde76298-df0e-49d0-927e-cd6d114e0320\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") "
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tmp-dir\") pod \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") "
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.042965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-model-cache\") pod \"fde76298-df0e-49d0-927e-cd6d114e0320\" (UID: \"fde76298-df0e-49d0-927e-cd6d114e0320\") "
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tls-certs\") pod \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") "
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-dshm\") pod \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\" (UID: \"b203b3b0-cc3c-4101-9570-16f7c6caed1d\") "
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043250    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-dshm\") pod \"04e53c54-d895-4d11-800b-c0017ca20e3d\" (UID: \"04e53c54-d895-4d11-800b-c0017ca20e3d\") "
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-dshm\") pod \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") "
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tmp-dir\") pod \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\" (UID: \"0a677d3a-7b7b-4c6d-b536-6a455103e7d7\") "
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043705    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043746    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043758    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.043769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043771    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.044622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043783    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.044622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043794    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.044622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043805    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.044622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043817    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-598dv\" (UniqueName: \"kubernetes.io/projected/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc-kube-api-access-598dv\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.044622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.043831    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.044622 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.044030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a677d3a-7b7b-4c6d-b536-6a455103e7d7" (UID: "0a677d3a-7b7b-4c6d-b536-6a455103e7d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.045356 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.045326    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fde76298-df0e-49d0-927e-cd6d114e0320" (UID: "fde76298-df0e-49d0-927e-cd6d114e0320"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.045602 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.045562    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "04e53c54-d895-4d11-800b-c0017ca20e3d" (UID: "04e53c54-d895-4d11-800b-c0017ca20e3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.045686 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.045643    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b203b3b0-cc3c-4101-9570-16f7c6caed1d" (UID: "b203b3b0-cc3c-4101-9570-16f7c6caed1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.045794 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.045773    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-home" (OuterVolumeSpecName: "home") pod "04e53c54-d895-4d11-800b-c0017ca20e3d" (UID: "04e53c54-d895-4d11-800b-c0017ca20e3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.045866 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.045808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a677d3a-7b7b-4c6d-b536-6a455103e7d7" (UID: "0a677d3a-7b7b-4c6d-b536-6a455103e7d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.045927 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.045886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-model-cache" (OuterVolumeSpecName: "model-cache") pod "fde76298-df0e-49d0-927e-cd6d114e0320" (UID: "fde76298-df0e-49d0-927e-cd6d114e0320"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.045927 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.045912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b203b3b0-cc3c-4101-9570-16f7c6caed1d-kube-api-access-5fznw" (OuterVolumeSpecName: "kube-api-access-5fznw") pod "b203b3b0-cc3c-4101-9570-16f7c6caed1d" (UID: "b203b3b0-cc3c-4101-9570-16f7c6caed1d"). InnerVolumeSpecName "kube-api-access-5fznw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:04.046208 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.046039    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-dshm" (OuterVolumeSpecName: "dshm") pod "fde76298-df0e-49d0-927e-cd6d114e0320" (UID: "fde76298-df0e-49d0-927e-cd6d114e0320"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.046208 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.046078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde76298-df0e-49d0-927e-cd6d114e0320-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fde76298-df0e-49d0-927e-cd6d114e0320" (UID: "fde76298-df0e-49d0-927e-cd6d114e0320"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:04.046560 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.046501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e53c54-d895-4d11-800b-c0017ca20e3d-kube-api-access-fp5lk" (OuterVolumeSpecName: "kube-api-access-fp5lk") pod "04e53c54-d895-4d11-800b-c0017ca20e3d" (UID: "04e53c54-d895-4d11-800b-c0017ca20e3d"). InnerVolumeSpecName "kube-api-access-fp5lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:04.046657 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.046555    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-dshm" (OuterVolumeSpecName: "dshm") pod "0a677d3a-7b7b-4c6d-b536-6a455103e7d7" (UID: "0a677d3a-7b7b-4c6d-b536-6a455103e7d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.046865 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.046846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e53c54-d895-4d11-800b-c0017ca20e3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04e53c54-d895-4d11-800b-c0017ca20e3d" (UID: "04e53c54-d895-4d11-800b-c0017ca20e3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:04.047306 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.047286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a677d3a-7b7b-4c6d-b536-6a455103e7d7" (UID: "0a677d3a-7b7b-4c6d-b536-6a455103e7d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:04.047770 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.047745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-kube-api-access-2crjl" (OuterVolumeSpecName: "kube-api-access-2crjl") pod "0a677d3a-7b7b-4c6d-b536-6a455103e7d7" (UID: "0a677d3a-7b7b-4c6d-b536-6a455103e7d7"). InnerVolumeSpecName "kube-api-access-2crjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:04.047980 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.047963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b203b3b0-cc3c-4101-9570-16f7c6caed1d" (UID: "b203b3b0-cc3c-4101-9570-16f7c6caed1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:04.048552 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.048537    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-dshm" (OuterVolumeSpecName: "dshm") pod "04e53c54-d895-4d11-800b-c0017ca20e3d" (UID: "04e53c54-d895-4d11-800b-c0017ca20e3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.048674 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.048659    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-dshm" (OuterVolumeSpecName: "dshm") pod "b203b3b0-cc3c-4101-9570-16f7c6caed1d" (UID: "b203b3b0-cc3c-4101-9570-16f7c6caed1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:04.048796 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.048681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde76298-df0e-49d0-927e-cd6d114e0320-kube-api-access-dlmnc" (OuterVolumeSpecName: "kube-api-access-dlmnc") pod "fde76298-df0e-49d0-927e-cd6d114e0320" (UID: "fde76298-df0e-49d0-927e-cd6d114e0320"). InnerVolumeSpecName "kube-api-access-dlmnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:04.119415 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.119379    2563 generic.go:356] "Generic (PLEG): container finished" podID="a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" containerID="7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0" exitCode=0
Sep 09 21:59:04.119602 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.119427    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"
Sep 09 21:59:04.119602 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.119476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj" event={"ID":"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc","Type":"ContainerDied","Data":"7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0"}
Sep 09 21:59:04.119602 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.119526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj" event={"ID":"a940c2ee-0fc5-4d46-9d97-0689d9f10ecc","Type":"ContainerDied","Data":"717050099d27c23b39d4e92550e82ffda981c47225eb5900681c921ac35dfd0b"}
Sep 09 21:59:04.119602 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.119555    2563 scope.go:117] "RemoveContainer" containerID="7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0"
Sep 09 21:59:04.120939 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.120910    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a677d3a-7b7b-4c6d-b536-6a455103e7d7" containerID="6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3" exitCode=0
Sep 09 21:59:04.121078 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.120942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7" event={"ID":"0a677d3a-7b7b-4c6d-b536-6a455103e7d7","Type":"ContainerDied","Data":"6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3"}
Sep 09 21:59:04.121078 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.120957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"
Sep 09 21:59:04.121078 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.120980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7" event={"ID":"0a677d3a-7b7b-4c6d-b536-6a455103e7d7","Type":"ContainerDied","Data":"f4db5ac855e3c40b00a0c3a9c17d06da81e82b40878e3bd4982cd821dbfa2761"}
Sep 09 21:59:04.122613 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.122588    2563 generic.go:356] "Generic (PLEG): container finished" podID="04e53c54-d895-4d11-800b-c0017ca20e3d" containerID="d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6" exitCode=0
Sep 09 21:59:04.122832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.122631    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"
Sep 09 21:59:04.122832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.122656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4" event={"ID":"04e53c54-d895-4d11-800b-c0017ca20e3d","Type":"ContainerDied","Data":"d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6"}
Sep 09 21:59:04.122832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.122687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4" event={"ID":"04e53c54-d895-4d11-800b-c0017ca20e3d","Type":"ContainerDied","Data":"f4f801db8499fc8447860c1e2616fa66f95814a893a16e162a269ba2c21c2d0a"}
Sep 09 21:59:04.133793 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.132443    2563 generic.go:356] "Generic (PLEG): container finished" podID="b203b3b0-cc3c-4101-9570-16f7c6caed1d" containerID="b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408" exitCode=0
Sep 09 21:59:04.133793 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.132640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"
Sep 09 21:59:04.133793 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.133452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz" event={"ID":"b203b3b0-cc3c-4101-9570-16f7c6caed1d","Type":"ContainerDied","Data":"b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408"}
Sep 09 21:59:04.133793 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.133490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz" event={"ID":"b203b3b0-cc3c-4101-9570-16f7c6caed1d","Type":"ContainerDied","Data":"c8329838d5ce2738c47541f7ea92ba154f42289432c3b4e5efbd74603a530d37"}
Sep 09 21:59:04.138042 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.138017    2563 generic.go:356] "Generic (PLEG): container finished" podID="fde76298-df0e-49d0-927e-cd6d114e0320" containerID="1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53" exitCode=0
Sep 09 21:59:04.138142 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.138066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr" event={"ID":"fde76298-df0e-49d0-927e-cd6d114e0320","Type":"ContainerDied","Data":"1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53"}
Sep 09 21:59:04.138228 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.138092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr" event={"ID":"fde76298-df0e-49d0-927e-cd6d114e0320","Type":"ContainerDied","Data":"25ec7d2e6bbc5a4da7f80f08918705e070804fd8c12373ea03157b24793a1d03"}
Sep 09 21:59:04.138228 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.138195    2563 scope.go:117] "RemoveContainer" containerID="7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0"
Sep 09 21:59:04.138324 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.138279    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"
Sep 09 21:59:04.138553 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:59:04.138534    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0\": container with ID starting with 7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0 not found: ID does not exist" containerID="7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0"
Sep 09 21:59:04.138616 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.138565    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0"} err="failed to get container status \"7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0\": rpc error: code = NotFound desc = could not find container \"7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0\": container with ID starting with 7617a8164bca321066492930fdac860b411971b755375c2dc2b06624e27d13b0 not found: ID does not exist"
Sep 09 21:59:04.138616 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.138585    2563 scope.go:117] "RemoveContainer" containerID="6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3"
Sep 09 21:59:04.144618 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144512    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e53c54-d895-4d11-800b-c0017ca20e3d-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.144618 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144562    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp5lk\" (UniqueName: \"kubernetes.io/projected/04e53c54-d895-4d11-800b-c0017ca20e3d-kube-api-access-fp5lk\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.144618 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144579    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlmnc\" (UniqueName: \"kubernetes.io/projected/fde76298-df0e-49d0-927e-cd6d114e0320-kube-api-access-dlmnc\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.144889 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144594    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde76298-df0e-49d0-927e-cd6d114e0320-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144898    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2crjl\" (UniqueName: \"kubernetes.io/projected/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-kube-api-access-2crjl\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144915    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144930    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144945    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144958    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fznw\" (UniqueName: \"kubernetes.io/projected/b203b3b0-cc3c-4101-9570-16f7c6caed1d-kube-api-access-5fznw\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144970    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144983    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.144994    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.145003    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.145014    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde76298-df0e-49d0-927e-cd6d114e0320-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145396 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.145025    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b203b3b0-cc3c-4101-9570-16f7c6caed1d-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145396 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.145037    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b203b3b0-cc3c-4101-9570-16f7c6caed1d-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145396 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.145047    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e53c54-d895-4d11-800b-c0017ca20e3d-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145396 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.145058    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.145396 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.145069    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a677d3a-7b7b-4c6d-b536-6a455103e7d7-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:04.149317 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.149297    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"]
Sep 09 21:59:04.150317 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.150304    2563 scope.go:117] "RemoveContainer" containerID="6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3"
Sep 09 21:59:04.150632 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:59:04.150616    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3\": container with ID starting with 6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3 not found: ID does not exist" containerID="6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3"
Sep 09 21:59:04.150673 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.150639    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3"} err="failed to get container status \"6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3\": rpc error: code = NotFound desc = could not find container \"6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3\": container with ID starting with 6bb572c21ba9dc8bc99d663a4ff6892688e72f09de0698ba8b78c0ea65f6d4a3 not found: ID does not exist"
Sep 09 21:59:04.150673 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.150658    2563 scope.go:117] "RemoveContainer" containerID="d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6"
Sep 09 21:59:04.155646 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.155623    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-w0-390abc-kserve-8445b5dfc5-h4zfj"]
Sep 09 21:59:04.159024 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.159008    2563 scope.go:117] "RemoveContainer" containerID="d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6"
Sep 09 21:59:04.159313 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:59:04.159295    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6\": container with ID starting with d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6 not found: ID does not exist" containerID="d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6"
Sep 09 21:59:04.159390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.159323    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6"} err="failed to get container status \"d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6\": rpc error: code = NotFound desc = could not find container \"d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6\": container with ID starting with d9baa112c78400ee15a6edcc32b0943685407d32cf07b324fe302db7ee784db6 not found: ID does not exist"
Sep 09 21:59:04.159390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.159344    2563 scope.go:117] "RemoveContainer" containerID="b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408"
Sep 09 21:59:04.167430 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.167414    2563 scope.go:117] "RemoveContainer" containerID="b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408"
Sep 09 21:59:04.167707 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:59:04.167688    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408\": container with ID starting with b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408 not found: ID does not exist" containerID="b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408"
Sep 09 21:59:04.167820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.167742    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408"} err="failed to get container status \"b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408\": rpc error: code = NotFound desc = could not find container \"b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408\": container with ID starting with b73c36c87f818563713401c44908161f4b76a616e86e76a2a1aeadb9714a5408 not found: ID does not exist"
Sep 09 21:59:04.167820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.167765    2563 scope.go:117] "RemoveContainer" containerID="1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53"
Sep 09 21:59:04.169873 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.169854    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"]
Sep 09 21:59:04.171906 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.171886    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/distinct-2-w0-390abc-kserve-6cf6f7fc8d-9kfj7"]
Sep 09 21:59:04.176247 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.176231    2563 scope.go:117] "RemoveContainer" containerID="1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53"
Sep 09 21:59:04.176540 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:59:04.176523    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53\": container with ID starting with 1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53 not found: ID does not exist" containerID="1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53"
Sep 09 21:59:04.176604 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.176545    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53"} err="failed to get container status \"1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53\": rpc error: code = NotFound desc = could not find container \"1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53\": container with ID starting with 1f5f3e2c63143578ca998efbbd66a68b1bf2fa4e4209c94d11066527f7ea6b53 not found: ID does not exist"
Sep 09 21:59:04.186896 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.186866    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"]
Sep 09 21:59:04.191097 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.191077    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/unconfigured-w0-390abc-kserve-b7584874d-8nfrz"]
Sep 09 21:59:04.201870 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.201841    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"]
Sep 09 21:59:04.203627 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.203608    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/premium-w0-390abc-kserve-88d577dd-ztgnr"]
Sep 09 21:59:04.214519 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.214497    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"]
Sep 09 21:59:04.218642 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:04.218621    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-390abc/simulator-w0-390abc-kserve-685b9f47bf-6czs4"]
Sep 09 21:59:05.047220 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.047190    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e53c54-d895-4d11-800b-c0017ca20e3d" path="/var/lib/kubelet/pods/04e53c54-d895-4d11-800b-c0017ca20e3d/volumes"
Sep 09 21:59:05.047592 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.047564    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a677d3a-7b7b-4c6d-b536-6a455103e7d7" path="/var/lib/kubelet/pods/0a677d3a-7b7b-4c6d-b536-6a455103e7d7/volumes"
Sep 09 21:59:05.047879 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.047865    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ef55f77-dfa1-49f4-a410-03efac73028d" path="/var/lib/kubelet/pods/4ef55f77-dfa1-49f4-a410-03efac73028d/volumes"
Sep 09 21:59:05.048508 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.048494    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" path="/var/lib/kubelet/pods/a940c2ee-0fc5-4d46-9d97-0689d9f10ecc/volumes"
Sep 09 21:59:05.048887 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.048875    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b203b3b0-cc3c-4101-9570-16f7c6caed1d" path="/var/lib/kubelet/pods/b203b3b0-cc3c-4101-9570-16f7c6caed1d/volumes"
Sep 09 21:59:05.049215 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.049203    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fde76298-df0e-49d0-927e-cd6d114e0320" path="/var/lib/kubelet/pods/fde76298-df0e-49d0-927e-cd6d114e0320/volumes"
Sep 09 21:59:05.712319 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.712284    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:59:05.712534 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.712336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:59:05.724990 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:05.724958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:59:06.166392 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:06.166357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 21:59:10.149008 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.148970    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"]
Sep 09 21:59:10.149379 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149366    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a677d3a-7b7b-4c6d-b536-6a455103e7d7" containerName="main"
Sep 09 21:59:10.149379 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149377    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a677d3a-7b7b-4c6d-b536-6a455103e7d7" containerName="main"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149386    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" containerName="main"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149391    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" containerName="main"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149410    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde76298-df0e-49d0-927e-cd6d114e0320" containerName="main"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149415    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde76298-df0e-49d0-927e-cd6d114e0320" containerName="main"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149423    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e53c54-d895-4d11-800b-c0017ca20e3d" containerName="main"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149429    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e53c54-d895-4d11-800b-c0017ca20e3d" containerName="main"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149443    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff02b24-2e79-461a-a09c-1d267306d5a6" containerName="payload-processing"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149448    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff02b24-2e79-461a-a09c-1d267306d5a6" containerName="payload-processing"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149455    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b203b3b0-cc3c-4101-9570-16f7c6caed1d" containerName="main"
Sep 09 21:59:10.149458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149461    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b203b3b0-cc3c-4101-9570-16f7c6caed1d" containerName="main"
Sep 09 21:59:10.149769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149521    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a677d3a-7b7b-4c6d-b536-6a455103e7d7" containerName="main"
Sep 09 21:59:10.149769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149531    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e53c54-d895-4d11-800b-c0017ca20e3d" containerName="main"
Sep 09 21:59:10.149769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149537    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a940c2ee-0fc5-4d46-9d97-0689d9f10ecc" containerName="main"
Sep 09 21:59:10.149769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149544    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde76298-df0e-49d0-927e-cd6d114e0320" containerName="main"
Sep 09 21:59:10.149769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149549    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b203b3b0-cc3c-4101-9570-16f7c6caed1d" containerName="main"
Sep 09 21:59:10.149769 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.149555    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff02b24-2e79-461a-a09c-1d267306d5a6" containerName="payload-processing"
Sep 09 21:59:10.154497 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.154471    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"
Sep 09 21:59:10.157149 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.157127    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:59:10.170859 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.170836    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"]
Sep 09 21:59:10.303223 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.303188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1063a1bc-0f64-455a-9a5d-c4b9423b0796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq\" (UID: \"1063a1bc-0f64-455a-9a5d-c4b9423b0796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"
Sep 09 21:59:10.404233 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.404144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1063a1bc-0f64-455a-9a5d-c4b9423b0796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq\" (UID: \"1063a1bc-0f64-455a-9a5d-c4b9423b0796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"
Sep 09 21:59:10.404786 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.404765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1063a1bc-0f64-455a-9a5d-c4b9423b0796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq\" (UID: \"1063a1bc-0f64-455a-9a5d-c4b9423b0796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"
Sep 09 21:59:10.464931 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.464901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"
Sep 09 21:59:10.755129 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:10.755042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"]
Sep 09 21:59:11.185107 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:11.185073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq" event={"ID":"1063a1bc-0f64-455a-9a5d-c4b9423b0796","Type":"ContainerStarted","Data":"221b728f9aa3cc22e87e75230b614a2c69eac120c5c45d95ec81b4ce3ad3529c"}
Sep 09 21:59:11.185518 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:11.185135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq" event={"ID":"1063a1bc-0f64-455a-9a5d-c4b9423b0796","Type":"ContainerStarted","Data":"f9362e6b1c6d0a2a7d852a7941dced4ba6ec427d2c1ac7202d6c5ba1ab1e85bb"}
Sep 09 21:59:11.228356 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:11.228298    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq" podStartSLOduration=1.228283905 podStartE2EDuration="1.228283905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:11.226656739 +0000 UTC m=+1556.712539843" watchObservedRunningTime="2026-09-09 21:59:11.228283905 +0000 UTC m=+1556.714166943"
Sep 09 21:59:12.192391 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:12.192358    2563 generic.go:356] "Generic (PLEG): container finished" podID="1063a1bc-0f64-455a-9a5d-c4b9423b0796" containerID="221b728f9aa3cc22e87e75230b614a2c69eac120c5c45d95ec81b4ce3ad3529c" exitCode=0
Sep 09 21:59:12.192845 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:12.192395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq" event={"ID":"1063a1bc-0f64-455a-9a5d-c4b9423b0796","Type":"ContainerDied","Data":"221b728f9aa3cc22e87e75230b614a2c69eac120c5c45d95ec81b4ce3ad3529c"}
Sep 09 21:59:13.333914 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:13.333886    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"
Sep 09 21:59:13.435210 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:13.435177    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1063a1bc-0f64-455a-9a5d-c4b9423b0796-maas-api-service-ca\") pod \"1063a1bc-0f64-455a-9a5d-c4b9423b0796\" (UID: \"1063a1bc-0f64-455a-9a5d-c4b9423b0796\") "
Sep 09 21:59:13.435533 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:13.435511    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1063a1bc-0f64-455a-9a5d-c4b9423b0796-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1063a1bc-0f64-455a-9a5d-c4b9423b0796" (UID: "1063a1bc-0f64-455a-9a5d-c4b9423b0796"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:13.536359 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:13.536286    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1063a1bc-0f64-455a-9a5d-c4b9423b0796-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:14.205288 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:14.205262    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"
Sep 09 21:59:14.205491 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:14.205302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq" event={"ID":"1063a1bc-0f64-455a-9a5d-c4b9423b0796","Type":"ContainerDied","Data":"f9362e6b1c6d0a2a7d852a7941dced4ba6ec427d2c1ac7202d6c5ba1ab1e85bb"}
Sep 09 21:59:14.205491 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:14.205350    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9362e6b1c6d0a2a7d852a7941dced4ba6ec427d2c1ac7202d6c5ba1ab1e85bb"
Sep 09 21:59:25.807875 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:25.807824    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"]
Sep 09 21:59:25.808363 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:25.808165    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8" podUID="4f30fe77-08f9-4804-9f76-1258a5fd077c" containerName="payload-processing" containerID="cri-o://139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228" gracePeriod=30
Sep 09 21:59:26.056349 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.056323    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:59:26.153295 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.153256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f30fe77-08f9-4804-9f76-1258a5fd077c-plugins-config-volume\") pod \"4f30fe77-08f9-4804-9f76-1258a5fd077c\" (UID: \"4f30fe77-08f9-4804-9f76-1258a5fd077c\") "
Sep 09 21:59:26.153493 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.153360    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgdwc\" (UniqueName: \"kubernetes.io/projected/4f30fe77-08f9-4804-9f76-1258a5fd077c-kube-api-access-mgdwc\") pod \"4f30fe77-08f9-4804-9f76-1258a5fd077c\" (UID: \"4f30fe77-08f9-4804-9f76-1258a5fd077c\") "
Sep 09 21:59:26.153660 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.153636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f30fe77-08f9-4804-9f76-1258a5fd077c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f30fe77-08f9-4804-9f76-1258a5fd077c" (UID: "4f30fe77-08f9-4804-9f76-1258a5fd077c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:26.155472 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.155447    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f30fe77-08f9-4804-9f76-1258a5fd077c-kube-api-access-mgdwc" (OuterVolumeSpecName: "kube-api-access-mgdwc") pod "4f30fe77-08f9-4804-9f76-1258a5fd077c" (UID: "4f30fe77-08f9-4804-9f76-1258a5fd077c"). InnerVolumeSpecName "kube-api-access-mgdwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:26.196906 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.196875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"]
Sep 09 21:59:26.201697 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.201674    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dedad-tp4qq"]
Sep 09 21:59:26.254461 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.254424    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgdwc\" (UniqueName: \"kubernetes.io/projected/4f30fe77-08f9-4804-9f76-1258a5fd077c-kube-api-access-mgdwc\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:26.254461 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.254457    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f30fe77-08f9-4804-9f76-1258a5fd077c-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:26.274309 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.274274    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f30fe77-08f9-4804-9f76-1258a5fd077c" containerID="139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228" exitCode=0
Sep 09 21:59:26.274458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.274322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"
Sep 09 21:59:26.274458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.274364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8" event={"ID":"4f30fe77-08f9-4804-9f76-1258a5fd077c","Type":"ContainerDied","Data":"139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228"}
Sep 09 21:59:26.274458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.274432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8" event={"ID":"4f30fe77-08f9-4804-9f76-1258a5fd077c","Type":"ContainerDied","Data":"7bd9259c672adbd2ec07c3ac6d6deb83a132df89dbdd85912c92b32b19f7693c"}
Sep 09 21:59:26.274458 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.274457    2563 scope.go:117] "RemoveContainer" containerID="139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228"
Sep 09 21:59:26.284084 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.284063    2563 scope.go:117] "RemoveContainer" containerID="139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228"
Sep 09 21:59:26.284357 ip-10-0-133-204 kubenswrapper[2563]: E0909 21:59:26.284334    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228\": container with ID starting with 139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228 not found: ID does not exist" containerID="139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228"
Sep 09 21:59:26.284457 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.284363    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228"} err="failed to get container status \"139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228\": rpc error: code = NotFound desc = could not find container \"139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228\": container with ID starting with 139569f664809f793297c73f61e502449820de77fa5b5b77d3806b2942db6228 not found: ID does not exist"
Sep 09 21:59:26.303527 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.303495    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"]
Sep 09 21:59:26.308365 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:26.308339    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9dedad-7d6b486d87-n4sk8"]
Sep 09 21:59:27.046357 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:27.046328    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1063a1bc-0f64-455a-9a5d-c4b9423b0796" path="/var/lib/kubelet/pods/1063a1bc-0f64-455a-9a5d-c4b9423b0796/volumes"
Sep 09 21:59:27.046733 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:27.046695    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f30fe77-08f9-4804-9f76-1258a5fd077c" path="/var/lib/kubelet/pods/4f30fe77-08f9-4804-9f76-1258a5fd077c/volumes"
Sep 09 21:59:34.370327 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.370292    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"]
Sep 09 21:59:34.370848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.370703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f30fe77-08f9-4804-9f76-1258a5fd077c" containerName="payload-processing"
Sep 09 21:59:34.370848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.370730    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f30fe77-08f9-4804-9f76-1258a5fd077c" containerName="payload-processing"
Sep 09 21:59:34.370848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.370740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1063a1bc-0f64-455a-9a5d-c4b9423b0796" containerName="revoke-keys"
Sep 09 21:59:34.370848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.370746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1063a1bc-0f64-455a-9a5d-c4b9423b0796" containerName="revoke-keys"
Sep 09 21:59:34.370848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.370822    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1063a1bc-0f64-455a-9a5d-c4b9423b0796" containerName="revoke-keys"
Sep 09 21:59:34.370848 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.370832    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f30fe77-08f9-4804-9f76-1258a5fd077c" containerName="payload-processing"
Sep 09 21:59:34.380423 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.380387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"
Sep 09 21:59:34.387210 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.387160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:59:34.388517 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.388495    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"]
Sep 09 21:59:34.438712 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.438677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59d74a6-f613-4f7b-bbc3-c3515f6b9b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4\" (UID: \"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"
Sep 09 21:59:34.539998 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.539957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59d74a6-f613-4f7b-bbc3-c3515f6b9b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4\" (UID: \"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"
Sep 09 21:59:34.540583 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.540561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59d74a6-f613-4f7b-bbc3-c3515f6b9b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4\" (UID: \"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"
Sep 09 21:59:34.693024 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.692935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"
Sep 09 21:59:34.976377 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:34.976351    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"]
Sep 09 21:59:34.978485 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:59:34.978442    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59d74a6_f613_4f7b_bbc3_c3515f6b9b16.slice/crio-5314c6c423b87ff6043d4ffb512f5f4bf64deed32958b245f6f5cc714a8957f5 WatchSource:0}: Error finding container 5314c6c423b87ff6043d4ffb512f5f4bf64deed32958b245f6f5cc714a8957f5: Status 404 returned error can't find the container with id 5314c6c423b87ff6043d4ffb512f5f4bf64deed32958b245f6f5cc714a8957f5
Sep 09 21:59:35.323913 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:35.323877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4" event={"ID":"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16","Type":"ContainerStarted","Data":"71e58621b119ce94fd8bc01b1e5ec5db9a8e45a35bc01228c636ef5df33d0a70"}
Sep 09 21:59:35.324038 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:35.323926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4" event={"ID":"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16","Type":"ContainerStarted","Data":"5314c6c423b87ff6043d4ffb512f5f4bf64deed32958b245f6f5cc714a8957f5"}
Sep 09 21:59:35.366390 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:35.366336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4" podStartSLOduration=1.3663198269999999 podStartE2EDuration="1.366319827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:35.363686999 +0000 UTC m=+1580.849570036" watchObservedRunningTime="2026-09-09 21:59:35.366319827 +0000 UTC m=+1580.852202865"
Sep 09 21:59:36.330820 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:36.330782    2563 generic.go:356] "Generic (PLEG): container finished" podID="f59d74a6-f613-4f7b-bbc3-c3515f6b9b16" containerID="71e58621b119ce94fd8bc01b1e5ec5db9a8e45a35bc01228c636ef5df33d0a70" exitCode=0
Sep 09 21:59:36.331234 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:36.330870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4" event={"ID":"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16","Type":"ContainerDied","Data":"71e58621b119ce94fd8bc01b1e5ec5db9a8e45a35bc01228c636ef5df33d0a70"}
Sep 09 21:59:37.476621 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:37.476595    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"
Sep 09 21:59:37.571122 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:37.571090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59d74a6-f613-4f7b-bbc3-c3515f6b9b16-maas-api-service-ca\") pod \"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16\" (UID: \"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16\") "
Sep 09 21:59:37.571457 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:37.571435    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59d74a6-f613-4f7b-bbc3-c3515f6b9b16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f59d74a6-f613-4f7b-bbc3-c3515f6b9b16" (UID: "f59d74a6-f613-4f7b-bbc3-c3515f6b9b16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:37.672690 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:37.672654    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59d74a6-f613-4f7b-bbc3-c3515f6b9b16-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:38.343743 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:38.343678    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"
Sep 09 21:59:38.343743 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:38.343733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4" event={"ID":"f59d74a6-f613-4f7b-bbc3-c3515f6b9b16","Type":"ContainerDied","Data":"5314c6c423b87ff6043d4ffb512f5f4bf64deed32958b245f6f5cc714a8957f5"}
Sep 09 21:59:38.343951 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:38.343784    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5314c6c423b87ff6043d4ffb512f5f4bf64deed32958b245f6f5cc714a8957f5"
Sep 09 21:59:45.107223 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:45.107186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"]
Sep 09 21:59:45.111584 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:45.111551    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d99c83-5zbx4"]
Sep 09 21:59:47.046441 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:47.046408    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f59d74a6-f613-4f7b-bbc3-c3515f6b9b16" path="/var/lib/kubelet/pods/f59d74a6-f613-4f7b-bbc3-c3515f6b9b16/volumes"
Sep 09 21:59:52.827960 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:52.827923    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"]
Sep 09 21:59:52.828370 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:52.828357    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59d74a6-f613-4f7b-bbc3-c3515f6b9b16" containerName="revoke-keys"
Sep 09 21:59:52.828415 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:52.828371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59d74a6-f613-4f7b-bbc3-c3515f6b9b16" containerName="revoke-keys"
Sep 09 21:59:52.828447 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:52.828434    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f59d74a6-f613-4f7b-bbc3-c3515f6b9b16" containerName="revoke-keys"
Sep 09 21:59:52.831292 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:52.831273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"
Sep 09 21:59:52.834129 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:52.834091    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 21:59:52.841824 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:52.841750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"]
Sep 09 21:59:53.024233 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:53.024188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28e660a-d034-4f07-abe4-1cd8df643e99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh\" (UID: \"a28e660a-d034-4f07-abe4-1cd8df643e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"
Sep 09 21:59:53.125450 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:53.125408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28e660a-d034-4f07-abe4-1cd8df643e99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh\" (UID: \"a28e660a-d034-4f07-abe4-1cd8df643e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"
Sep 09 21:59:53.126074 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:53.126051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28e660a-d034-4f07-abe4-1cd8df643e99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh\" (UID: \"a28e660a-d034-4f07-abe4-1cd8df643e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"
Sep 09 21:59:53.142261 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:53.142241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"
Sep 09 21:59:53.419470 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:53.419446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"]
Sep 09 21:59:53.421168 ip-10-0-133-204 kubenswrapper[2563]: W0909 21:59:53.421142    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28e660a_d034_4f07_abe4_1cd8df643e99.slice/crio-fd43015d3d40e2aa6f7ad1d4b5e2faae6017202bf3b49203df0bd6b7beba7aaa WatchSource:0}: Error finding container fd43015d3d40e2aa6f7ad1d4b5e2faae6017202bf3b49203df0bd6b7beba7aaa: Status 404 returned error can't find the container with id fd43015d3d40e2aa6f7ad1d4b5e2faae6017202bf3b49203df0bd6b7beba7aaa
Sep 09 21:59:54.432166 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:54.432078    2563 generic.go:356] "Generic (PLEG): container finished" podID="a28e660a-d034-4f07-abe4-1cd8df643e99" containerID="b77816bfba2d5c647b0be56ba056d9be0b314e9deeeee792bde6663b5e4a0252" exitCode=0
Sep 09 21:59:54.432522 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:54.432205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh" event={"ID":"a28e660a-d034-4f07-abe4-1cd8df643e99","Type":"ContainerDied","Data":"b77816bfba2d5c647b0be56ba056d9be0b314e9deeeee792bde6663b5e4a0252"}
Sep 09 21:59:54.432522 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:54.432260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh" event={"ID":"a28e660a-d034-4f07-abe4-1cd8df643e99","Type":"ContainerStarted","Data":"fd43015d3d40e2aa6f7ad1d4b5e2faae6017202bf3b49203df0bd6b7beba7aaa"}
Sep 09 21:59:54.991428 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:54.991390    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"]
Sep 09 21:59:54.995653 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:54.995630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:54.998711 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:54.998684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fa986\"/\"e2e-conflict-b-d5ffdfe0-kserve-self-signed-certs\""
Sep 09 21:59:55.003240 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.003216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"]
Sep 09 21:59:55.147682 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.147643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-home\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.147682 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.147687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-model-cache\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.147988 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.147816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-tmp-dir\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.147988 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.147852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a372ca85-3400-428d-b76c-6076f31aa1ab-tls-certs\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.148111 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.148006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-dshm\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.148111 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.148047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8mdr\" (UniqueName: \"kubernetes.io/projected/a372ca85-3400-428d-b76c-6076f31aa1ab-kube-api-access-b8mdr\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249100 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-dshm\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249100 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8mdr\" (UniqueName: \"kubernetes.io/projected/a372ca85-3400-428d-b76c-6076f31aa1ab-kube-api-access-b8mdr\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249100 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-home\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249387 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-model-cache\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249387 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-tmp-dir\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249387 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a372ca85-3400-428d-b76c-6076f31aa1ab-tls-certs\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-home\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249562 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-model-cache\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.249644 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.249591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-tmp-dir\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.251418 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.251390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-dshm\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.251746 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.251730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a372ca85-3400-428d-b76c-6076f31aa1ab-tls-certs\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.258832 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.258809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8mdr\" (UniqueName: \"kubernetes.io/projected/a372ca85-3400-428d-b76c-6076f31aa1ab-kube-api-access-b8mdr\") pod \"e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") " pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.306395 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.306360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 21:59:55.568401 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.568376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"
Sep 09 21:59:55.581478 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.581445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"]
Sep 09 21:59:55.653025 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.653002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28e660a-d034-4f07-abe4-1cd8df643e99-maas-api-service-ca\") pod \"a28e660a-d034-4f07-abe4-1cd8df643e99\" (UID: \"a28e660a-d034-4f07-abe4-1cd8df643e99\") "
Sep 09 21:59:55.653447 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.653420    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28e660a-d034-4f07-abe4-1cd8df643e99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a28e660a-d034-4f07-abe4-1cd8df643e99" (UID: "a28e660a-d034-4f07-abe4-1cd8df643e99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:55.754680 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:55.754647    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28e660a-d034-4f07-abe4-1cd8df643e99-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 21:59:56.444949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:56.444909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z" event={"ID":"a372ca85-3400-428d-b76c-6076f31aa1ab","Type":"ContainerStarted","Data":"fc6d611372bb9e2c912e0807874c3702aa9e0f122547ba556c3978775600ccda"}
Sep 09 21:59:56.444949 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:56.444961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z" event={"ID":"a372ca85-3400-428d-b76c-6076f31aa1ab","Type":"ContainerStarted","Data":"f10e5584fa22aa3daf0bd12a4d84c368574cfadc4b2a8e425e3bab4b7ec33c51"}
Sep 09 21:59:56.446263 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:56.446240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh" event={"ID":"a28e660a-d034-4f07-abe4-1cd8df643e99","Type":"ContainerDied","Data":"fd43015d3d40e2aa6f7ad1d4b5e2faae6017202bf3b49203df0bd6b7beba7aaa"}
Sep 09 21:59:56.446349 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:56.446272    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"
Sep 09 21:59:56.446349 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:56.446280    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd43015d3d40e2aa6f7ad1d4b5e2faae6017202bf3b49203df0bd6b7beba7aaa"
Sep 09 21:59:56.464055 ip-10-0-133-204 kubenswrapper[2563]: I0909 21:59:56.464012    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z" podStartSLOduration=2.463997726 podStartE2EDuration="2.463997726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:56.463383443 +0000 UTC m=+1601.949266479" watchObservedRunningTime="2026-09-09 21:59:56.463997726 +0000 UTC m=+1601.949880764"
Sep 09 22:00:00.152472 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.152436    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"]
Sep 09 22:00:00.152958 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.152941    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28e660a-d034-4f07-abe4-1cd8df643e99" containerName="revoke-keys"
Sep 09 22:00:00.153019 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.152961    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28e660a-d034-4f07-abe4-1cd8df643e99" containerName="revoke-keys"
Sep 09 22:00:00.153058 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.153038    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28e660a-d034-4f07-abe4-1cd8df643e99" containerName="revoke-keys"
Sep 09 22:00:00.156314 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.156295    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"
Sep 09 22:00:00.160118 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.160095    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 22:00:00.163019 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.162995    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"]
Sep 09 22:00:00.166778 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.166756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"
Sep 09 22:00:00.167148 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.167131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"
Sep 09 22:00:00.174859 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.174836    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"]
Sep 09 22:00:00.178278 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.178108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"
Sep 09 22:00:00.180073 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.180018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"]
Sep 09 22:00:00.181532 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.180164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"
Sep 09 22:00:00.185440 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.185416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"]
Sep 09 22:00:00.190398 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.190290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"]
Sep 09 22:00:00.233779 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.233757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"
Sep 09 22:00:00.462675 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.462645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"]
Sep 09 22:00:00.464820 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:00:00.464793    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84efaf3_e314_4e70_bb30_a138d3b2f021.slice/crio-a0dcf847129386cc7e7c417de93d0a9473c78c66efcc8354659e3627e2422ef8 WatchSource:0}: Error finding container a0dcf847129386cc7e7c417de93d0a9473c78c66efcc8354659e3627e2422ef8: Status 404 returned error can't find the container with id a0dcf847129386cc7e7c417de93d0a9473c78c66efcc8354659e3627e2422ef8
Sep 09 22:00:00.476380 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.476357    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"]
Sep 09 22:00:00.479470 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:00:00.479446    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba96375_93f4_4ad2_8740_e78764da0dfc.slice/crio-76bd9892aa8d1bc8d535bee0e28b7686f41a84baba04f2d6bb0bd4a5d001376b WatchSource:0}: Error finding container 76bd9892aa8d1bc8d535bee0e28b7686f41a84baba04f2d6bb0bd4a5d001376b: Status 404 returned error can't find the container with id 76bd9892aa8d1bc8d535bee0e28b7686f41a84baba04f2d6bb0bd4a5d001376b
Sep 09 22:00:00.529134 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:00.529109    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"]
Sep 09 22:00:00.531107 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:00:00.531078    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6e7cc7_cc9f_476d_8b4d_2093330dbab8.slice/crio-013a88153becb966f90e2d8401f8cbcec9ee84b0e061d6099025fc584557d80e WatchSource:0}: Error finding container 013a88153becb966f90e2d8401f8cbcec9ee84b0e061d6099025fc584557d80e: Status 404 returned error can't find the container with id 013a88153becb966f90e2d8401f8cbcec9ee84b0e061d6099025fc584557d80e
Sep 09 22:00:01.475390 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.475360    2563 generic.go:356] "Generic (PLEG): container finished" podID="d84efaf3-e314-4e70-bb30-a138d3b2f021" containerID="165a36dec798ada525fba358ff09ed048285071417d7d2e9751c6045f44fbab1" exitCode=0
Sep 09 22:00:01.475901 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.475448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75" event={"ID":"d84efaf3-e314-4e70-bb30-a138d3b2f021","Type":"ContainerDied","Data":"165a36dec798ada525fba358ff09ed048285071417d7d2e9751c6045f44fbab1"}
Sep 09 22:00:01.475901 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.475508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75" event={"ID":"d84efaf3-e314-4e70-bb30-a138d3b2f021","Type":"ContainerStarted","Data":"a0dcf847129386cc7e7c417de93d0a9473c78c66efcc8354659e3627e2422ef8"}
Sep 09 22:00:01.477183 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.477158    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae6e7cc7-cc9f-476d-8b4d-2093330dbab8" containerID="eeeb8b6399e904af55cfecb8446395a0e7232e325d3d55cb25efe8ed157d5ad9" exitCode=0
Sep 09 22:00:01.477310 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.477280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8" event={"ID":"ae6e7cc7-cc9f-476d-8b4d-2093330dbab8","Type":"ContainerDied","Data":"eeeb8b6399e904af55cfecb8446395a0e7232e325d3d55cb25efe8ed157d5ad9"}
Sep 09 22:00:01.477372 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.477329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8" event={"ID":"ae6e7cc7-cc9f-476d-8b4d-2093330dbab8","Type":"ContainerStarted","Data":"013a88153becb966f90e2d8401f8cbcec9ee84b0e061d6099025fc584557d80e"}
Sep 09 22:00:01.478841 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.478817    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ba96375-93f4-4ad2-8740-e78764da0dfc" containerID="0ec2b69b4efa2f4406ad7caf9ca6779e7537ce78fba19a76a0e0b8e496c6d984" exitCode=0
Sep 09 22:00:01.478929 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.478854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp" event={"ID":"6ba96375-93f4-4ad2-8740-e78764da0dfc","Type":"ContainerDied","Data":"0ec2b69b4efa2f4406ad7caf9ca6779e7537ce78fba19a76a0e0b8e496c6d984"}
Sep 09 22:00:01.478929 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:01.478881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp" event={"ID":"6ba96375-93f4-4ad2-8740-e78764da0dfc","Type":"ContainerStarted","Data":"76bd9892aa8d1bc8d535bee0e28b7686f41a84baba04f2d6bb0bd4a5d001376b"}
Sep 09 22:00:02.635172 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:02.635145    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"
Sep 09 22:00:02.673851 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:02.673821    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"
Sep 09 22:00:02.677024 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:02.676997    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"
Sep 09 22:00:03.491214 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.491186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"
Sep 09 22:00:03.491214 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.491201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp" event={"ID":"6ba96375-93f4-4ad2-8740-e78764da0dfc","Type":"ContainerDied","Data":"76bd9892aa8d1bc8d535bee0e28b7686f41a84baba04f2d6bb0bd4a5d001376b"}
Sep 09 22:00:03.491443 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.491253    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76bd9892aa8d1bc8d535bee0e28b7686f41a84baba04f2d6bb0bd4a5d001376b"
Sep 09 22:00:03.492662 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.492639    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"
Sep 09 22:00:03.492662 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.492647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75" event={"ID":"d84efaf3-e314-4e70-bb30-a138d3b2f021","Type":"ContainerDied","Data":"a0dcf847129386cc7e7c417de93d0a9473c78c66efcc8354659e3627e2422ef8"}
Sep 09 22:00:03.492855 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.492691    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0dcf847129386cc7e7c417de93d0a9473c78c66efcc8354659e3627e2422ef8"
Sep 09 22:00:03.494225 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.494208    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"
Sep 09 22:00:03.494336 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.494264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8" event={"ID":"ae6e7cc7-cc9f-476d-8b4d-2093330dbab8","Type":"ContainerDied","Data":"013a88153becb966f90e2d8401f8cbcec9ee84b0e061d6099025fc584557d80e"}
Sep 09 22:00:03.494336 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.494304    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="013a88153becb966f90e2d8401f8cbcec9ee84b0e061d6099025fc584557d80e"
Sep 09 22:00:03.587409 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.587377    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"]
Sep 09 22:00:03.600392 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.600360    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9e275c-29816520-lhvmp"]
Sep 09 22:00:03.833056 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.832952    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"]
Sep 09 22:00:03.835641 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:03.835614    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e275c-7zlgh"]
Sep 09 22:00:05.046987 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:05.046952    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba96375-93f4-4ad2-8740-e78764da0dfc" path="/var/lib/kubelet/pods/6ba96375-93f4-4ad2-8740-e78764da0dfc/volumes"
Sep 09 22:00:05.047390 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:05.047237    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a28e660a-d034-4f07-abe4-1cd8df643e99" path="/var/lib/kubelet/pods/a28e660a-d034-4f07-abe4-1cd8df643e99/volumes"
Sep 09 22:00:05.306674 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:05.306573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 22:00:05.306674 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:05.306639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 22:00:05.319641 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:05.319614    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 22:00:05.516672 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:05.516601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 22:00:12.379828 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:12.379791    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"]
Sep 09 22:00:14.799407 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:14.799375    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"]
Sep 09 22:00:15.301952 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:15.301917    2563 scope.go:117] "RemoveContainer" containerID="00963a416b187765c1c9822ddb38f5e22053dcff63603fa1ab2b1e5ccd146e3e"
Sep 09 22:00:15.316590 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:15.316565    2563 scope.go:117] "RemoveContainer" containerID="e2cfdbb15e48d07e55c0529a34c45fe132090c5b706f997603f87ebeb464e2b6"
Sep 09 22:00:27.400157 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.400095    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z" podUID="a372ca85-3400-428d-b76c-6076f31aa1ab" containerName="main" containerID="cri-o://fc6d611372bb9e2c912e0807874c3702aa9e0f122547ba556c3978775600ccda" gracePeriod=45
Sep 09 22:00:27.631887 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.631857    2563 generic.go:356] "Generic (PLEG): container finished" podID="a372ca85-3400-428d-b76c-6076f31aa1ab" containerID="fc6d611372bb9e2c912e0807874c3702aa9e0f122547ba556c3978775600ccda" exitCode=0
Sep 09 22:00:27.632055 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.631934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z" event={"ID":"a372ca85-3400-428d-b76c-6076f31aa1ab","Type":"ContainerDied","Data":"fc6d611372bb9e2c912e0807874c3702aa9e0f122547ba556c3978775600ccda"}
Sep 09 22:00:27.657112 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.657052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 22:00:27.779365 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.779327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a372ca85-3400-428d-b76c-6076f31aa1ab-tls-certs\") pod \"a372ca85-3400-428d-b76c-6076f31aa1ab\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") "
Sep 09 22:00:27.779574 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.779382    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-tmp-dir\") pod \"a372ca85-3400-428d-b76c-6076f31aa1ab\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") "
Sep 09 22:00:27.779574 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.779399    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-dshm\") pod \"a372ca85-3400-428d-b76c-6076f31aa1ab\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") "
Sep 09 22:00:27.779574 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.779429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-home\") pod \"a372ca85-3400-428d-b76c-6076f31aa1ab\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") "
Sep 09 22:00:27.779574 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.779455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-model-cache\") pod \"a372ca85-3400-428d-b76c-6076f31aa1ab\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") "
Sep 09 22:00:27.779835 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.779602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8mdr\" (UniqueName: \"kubernetes.io/projected/a372ca85-3400-428d-b76c-6076f31aa1ab-kube-api-access-b8mdr\") pod \"a372ca85-3400-428d-b76c-6076f31aa1ab\" (UID: \"a372ca85-3400-428d-b76c-6076f31aa1ab\") "
Sep 09 22:00:27.779835 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.779686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a372ca85-3400-428d-b76c-6076f31aa1ab" (UID: "a372ca85-3400-428d-b76c-6076f31aa1ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:27.779835 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.779769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-home" (OuterVolumeSpecName: "home") pod "a372ca85-3400-428d-b76c-6076f31aa1ab" (UID: "a372ca85-3400-428d-b76c-6076f31aa1ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:27.780086 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.780065    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:27.780151 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.780091    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:27.780151 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.780081    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "a372ca85-3400-428d-b76c-6076f31aa1ab" (UID: "a372ca85-3400-428d-b76c-6076f31aa1ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:27.781980 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.781952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-dshm" (OuterVolumeSpecName: "dshm") pod "a372ca85-3400-428d-b76c-6076f31aa1ab" (UID: "a372ca85-3400-428d-b76c-6076f31aa1ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:27.782097 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.782036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a372ca85-3400-428d-b76c-6076f31aa1ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a372ca85-3400-428d-b76c-6076f31aa1ab" (UID: "a372ca85-3400-428d-b76c-6076f31aa1ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:00:27.782097 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.782048    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a372ca85-3400-428d-b76c-6076f31aa1ab-kube-api-access-b8mdr" (OuterVolumeSpecName: "kube-api-access-b8mdr") pod "a372ca85-3400-428d-b76c-6076f31aa1ab" (UID: "a372ca85-3400-428d-b76c-6076f31aa1ab"). InnerVolumeSpecName "kube-api-access-b8mdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:27.880919 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.880875    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8mdr\" (UniqueName: \"kubernetes.io/projected/a372ca85-3400-428d-b76c-6076f31aa1ab-kube-api-access-b8mdr\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:27.880919 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.880911    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a372ca85-3400-428d-b76c-6076f31aa1ab-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:27.880919 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.880920    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:27.880919 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:27.880928    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a372ca85-3400-428d-b76c-6076f31aa1ab-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:28.638496 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:28.638462    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"
Sep 09 22:00:28.639019 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:28.638494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z" event={"ID":"a372ca85-3400-428d-b76c-6076f31aa1ab","Type":"ContainerDied","Data":"f10e5584fa22aa3daf0bd12a4d84c368574cfadc4b2a8e425e3bab4b7ec33c51"}
Sep 09 22:00:28.639019 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:28.638557    2563 scope.go:117] "RemoveContainer" containerID="fc6d611372bb9e2c912e0807874c3702aa9e0f122547ba556c3978775600ccda"
Sep 09 22:00:28.663605 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:28.663578    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"]
Sep 09 22:00:28.665633 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:28.665613    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-b-d5ffdfe0-kserve-579f94584c-jf47z"]
Sep 09 22:00:29.046468 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:29.046391    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a372ca85-3400-428d-b76c-6076f31aa1ab" path="/var/lib/kubelet/pods/a372ca85-3400-428d-b76c-6076f31aa1ab/volumes"
Sep 09 22:00:29.819682 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:29.819632    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb" podUID="6cdf9dcc-148a-44df-a088-4c1836a57890" containerName="main" containerID="cri-o://9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b" gracePeriod=45
Sep 09 22:00:30.070612 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.070552    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 22:00:30.102055 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdf9dcc-148a-44df-a088-4c1836a57890-tls-certs\") pod \"6cdf9dcc-148a-44df-a088-4c1836a57890\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") "
Sep 09 22:00:30.102055 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-tmp-dir\") pod \"6cdf9dcc-148a-44df-a088-4c1836a57890\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") "
Sep 09 22:00:30.102294 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102092    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-home\") pod \"6cdf9dcc-148a-44df-a088-4c1836a57890\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") "
Sep 09 22:00:30.102294 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-model-cache\") pod \"6cdf9dcc-148a-44df-a088-4c1836a57890\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") "
Sep 09 22:00:30.102294 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102162    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-dshm\") pod \"6cdf9dcc-148a-44df-a088-4c1836a57890\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") "
Sep 09 22:00:30.102294 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn289\" (UniqueName: \"kubernetes.io/projected/6cdf9dcc-148a-44df-a088-4c1836a57890-kube-api-access-sn289\") pod \"6cdf9dcc-148a-44df-a088-4c1836a57890\" (UID: \"6cdf9dcc-148a-44df-a088-4c1836a57890\") "
Sep 09 22:00:30.102564 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102351    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cdf9dcc-148a-44df-a088-4c1836a57890" (UID: "6cdf9dcc-148a-44df-a088-4c1836a57890"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:30.102564 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-home" (OuterVolumeSpecName: "home") pod "6cdf9dcc-148a-44df-a088-4c1836a57890" (UID: "6cdf9dcc-148a-44df-a088-4c1836a57890"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:30.102564 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cdf9dcc-148a-44df-a088-4c1836a57890" (UID: "6cdf9dcc-148a-44df-a088-4c1836a57890"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:30.102709 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102622    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:30.102709 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102641    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:30.102709 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.102657    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:30.104326 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.104297    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdf9dcc-148a-44df-a088-4c1836a57890-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cdf9dcc-148a-44df-a088-4c1836a57890" (UID: "6cdf9dcc-148a-44df-a088-4c1836a57890"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:00:30.104443 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.104427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdf9dcc-148a-44df-a088-4c1836a57890-kube-api-access-sn289" (OuterVolumeSpecName: "kube-api-access-sn289") pod "6cdf9dcc-148a-44df-a088-4c1836a57890" (UID: "6cdf9dcc-148a-44df-a088-4c1836a57890"). InnerVolumeSpecName "kube-api-access-sn289". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:30.104681 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.104661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-dshm" (OuterVolumeSpecName: "dshm") pod "6cdf9dcc-148a-44df-a088-4c1836a57890" (UID: "6cdf9dcc-148a-44df-a088-4c1836a57890"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:30.203768 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.203739    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdf9dcc-148a-44df-a088-4c1836a57890-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:30.203768 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.203764    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdf9dcc-148a-44df-a088-4c1836a57890-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:30.203768 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.203772    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn289\" (UniqueName: \"kubernetes.io/projected/6cdf9dcc-148a-44df-a088-4c1836a57890-kube-api-access-sn289\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:00:30.651920 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.651874    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cdf9dcc-148a-44df-a088-4c1836a57890" containerID="9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b" exitCode=0
Sep 09 22:00:30.652143 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.651925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"
Sep 09 22:00:30.652143 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.651931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb" event={"ID":"6cdf9dcc-148a-44df-a088-4c1836a57890","Type":"ContainerDied","Data":"9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b"}
Sep 09 22:00:30.652143 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.652048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb" event={"ID":"6cdf9dcc-148a-44df-a088-4c1836a57890","Type":"ContainerDied","Data":"b93581730d7a9b2bc8c72b147e5f1a709f34d55c0a08ba8af15e550129c0f031"}
Sep 09 22:00:30.652143 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.652074    2563 scope.go:117] "RemoveContainer" containerID="9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b"
Sep 09 22:00:30.661841 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.661821    2563 scope.go:117] "RemoveContainer" containerID="9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b"
Sep 09 22:00:30.662161 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:00:30.662140    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b\": container with ID starting with 9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b not found: ID does not exist" containerID="9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b"
Sep 09 22:00:30.662266 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.662169    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b"} err="failed to get container status \"9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b\": rpc error: code = NotFound desc = could not find container \"9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b\": container with ID starting with 9a6cf801c4f9b45dd8540c103a42a6f8fe892c7ab055b34485fec82344a42d7b not found: ID does not exist"
Sep 09 22:00:30.673592 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.673564    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"]
Sep 09 22:00:30.677727 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:30.677691    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/e2e-conflict-a-d5ffdfe0-kserve-7cc55b95fc-6xqtb"]
Sep 09 22:00:31.047196 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:00:31.047107    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cdf9dcc-148a-44df-a088-4c1836a57890" path="/var/lib/kubelet/pods/6cdf9dcc-148a-44df-a088-4c1836a57890/volumes"
Sep 09 22:01:15.348635 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:15.348590    2563 scope.go:117] "RemoveContainer" containerID="6d19f0eb7f46a89e8c39779b11f9e206f58c185666cb89560da3dff03795c9a3"
Sep 09 22:01:33.211104 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:33.211070    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"]
Sep 09 22:01:40.682927 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.682840    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"]
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683245    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6e7cc7-cc9f-476d-8b4d-2093330dbab8" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683256    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6e7cc7-cc9f-476d-8b4d-2093330dbab8" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683265    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cdf9dcc-148a-44df-a088-4c1836a57890" containerName="main"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683270    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cdf9dcc-148a-44df-a088-4c1836a57890" containerName="main"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba96375-93f4-4ad2-8740-e78764da0dfc" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683282    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba96375-93f4-4ad2-8740-e78764da0dfc" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683295    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a372ca85-3400-428d-b76c-6076f31aa1ab" containerName="main"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683301    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a372ca85-3400-428d-b76c-6076f31aa1ab" containerName="main"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683311    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84efaf3-e314-4e70-bb30-a138d3b2f021" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683316    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84efaf3-e314-4e70-bb30-a138d3b2f021" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683383    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba96375-93f4-4ad2-8740-e78764da0dfc" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683390    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84efaf3-e314-4e70-bb30-a138d3b2f021" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683396    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6e7cc7-cc9f-476d-8b4d-2093330dbab8" containerName="cleanup"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683404    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a372ca85-3400-428d-b76c-6076f31aa1ab" containerName="main"
Sep 09 22:01:40.683402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.683409    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cdf9dcc-148a-44df-a088-4c1836a57890" containerName="main"
Sep 09 22:01:40.686627 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.686607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.689399 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.689374    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d3bc8\"/\"sub-test-model-2d3bc8-kserve-self-signed-certs\""
Sep 09 22:01:40.697863 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.697839    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"]
Sep 09 22:01:40.716658 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.716567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-dshm\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.716658 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.716660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tmp-dir\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.716953 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.716689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-model-cache\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.716953 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.716704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tls-certs\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.716953 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.716784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c78c\" (UniqueName: \"kubernetes.io/projected/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-kube-api-access-2c78c\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.716953 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.716871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-home\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.817755 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.817697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-dshm\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.817755 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.817758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tmp-dir\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.818017 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.817788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-model-cache\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.818017 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.817809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tls-certs\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.818017 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.817854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c78c\" (UniqueName: \"kubernetes.io/projected/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-kube-api-access-2c78c\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.818017 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.817927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-home\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.818241 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.818217    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-model-cache\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.818309 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.818278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tmp-dir\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.818365 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.818314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-home\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.820243 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.820220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-dshm\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.820468 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.820449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tls-certs\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.826443 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.826414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c78c\" (UniqueName: \"kubernetes.io/projected/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-kube-api-access-2c78c\") pod \"sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") " pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:40.997201 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:40.997113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:41.270779 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:41.270748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"]
Sep 09 22:01:41.273081 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:01:41.273046    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32ef9aca_f1d3_463a_a12e_adc2aaa04b0f.slice/crio-2096e6311d80dd5b420eef07a79095ff040e8589bd4d8706a90cdb0fa7a525e6 WatchSource:0}: Error finding container 2096e6311d80dd5b420eef07a79095ff040e8589bd4d8706a90cdb0fa7a525e6: Status 404 returned error can't find the container with id 2096e6311d80dd5b420eef07a79095ff040e8589bd4d8706a90cdb0fa7a525e6
Sep 09 22:01:42.052463 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:42.052426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc" event={"ID":"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f","Type":"ContainerStarted","Data":"7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a"}
Sep 09 22:01:42.052463 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:42.052468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc" event={"ID":"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f","Type":"ContainerStarted","Data":"2096e6311d80dd5b420eef07a79095ff040e8589bd4d8706a90cdb0fa7a525e6"}
Sep 09 22:01:42.069895 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:42.069843    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc" podStartSLOduration=2.069828525 podStartE2EDuration="2.069828525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:01:42.068210787 +0000 UTC m=+1707.554093826" watchObservedRunningTime="2026-09-09 22:01:42.069828525 +0000 UTC m=+1707.555711563"
Sep 09 22:01:48.231313 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.231248    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4" podUID="5b284dfd-a5c5-4199-8222-f30ee1bd62b4" containerName="main" containerID="cri-o://ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718" gracePeriod=45
Sep 09 22:01:48.489194 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.489128    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 22:01:48.592957 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.592917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-dshm\") pod \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") "
Sep 09 22:01:48.592957 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.592967    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-model-cache\") pod \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") "
Sep 09 22:01:48.593196 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-home\") pod \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") "
Sep 09 22:01:48.593196 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593048    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tmp-dir\") pod \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") "
Sep 09 22:01:48.593196 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-467fv\" (UniqueName: \"kubernetes.io/projected/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-kube-api-access-467fv\") pod \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") "
Sep 09 22:01:48.593196 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tls-certs\") pod \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\" (UID: \"5b284dfd-a5c5-4199-8222-f30ee1bd62b4\") "
Sep 09 22:01:48.593403 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-home" (OuterVolumeSpecName: "home") pod "5b284dfd-a5c5-4199-8222-f30ee1bd62b4" (UID: "5b284dfd-a5c5-4199-8222-f30ee1bd62b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:01:48.593403 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b284dfd-a5c5-4199-8222-f30ee1bd62b4" (UID: "5b284dfd-a5c5-4199-8222-f30ee1bd62b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:01:48.593511 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593406    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b284dfd-a5c5-4199-8222-f30ee1bd62b4" (UID: "5b284dfd-a5c5-4199-8222-f30ee1bd62b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:01:48.593607 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593590    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:01:48.593663 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593611    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:01:48.593663 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.593626    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:01:48.595405 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.595382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-dshm" (OuterVolumeSpecName: "dshm") pod "5b284dfd-a5c5-4199-8222-f30ee1bd62b4" (UID: "5b284dfd-a5c5-4199-8222-f30ee1bd62b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:01:48.595501 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.595467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b284dfd-a5c5-4199-8222-f30ee1bd62b4" (UID: "5b284dfd-a5c5-4199-8222-f30ee1bd62b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:01:48.595559 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.595523    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-kube-api-access-467fv" (OuterVolumeSpecName: "kube-api-access-467fv") pod "5b284dfd-a5c5-4199-8222-f30ee1bd62b4" (UID: "5b284dfd-a5c5-4199-8222-f30ee1bd62b4"). InnerVolumeSpecName "kube-api-access-467fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:01:48.695115 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.695083    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:01:48.695115 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.695111    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:01:48.695115 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:48.695120    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-467fv\" (UniqueName: \"kubernetes.io/projected/5b284dfd-a5c5-4199-8222-f30ee1bd62b4-kube-api-access-467fv\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:01:49.093329 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.093296    2563 generic.go:356] "Generic (PLEG): container finished" podID="5b284dfd-a5c5-4199-8222-f30ee1bd62b4" containerID="ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718" exitCode=0
Sep 09 22:01:49.093499 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.093337    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"
Sep 09 22:01:49.093499 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.093386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4" event={"ID":"5b284dfd-a5c5-4199-8222-f30ee1bd62b4","Type":"ContainerDied","Data":"ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718"}
Sep 09 22:01:49.093499 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.093424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4" event={"ID":"5b284dfd-a5c5-4199-8222-f30ee1bd62b4","Type":"ContainerDied","Data":"c7dd71bb4dc980673a57be8e2687ac6a1ebff72751e0bac2df78a6ba3e501d64"}
Sep 09 22:01:49.093499 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.093441    2563 scope.go:117] "RemoveContainer" containerID="ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718"
Sep 09 22:01:49.102303 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.102285    2563 scope.go:117] "RemoveContainer" containerID="ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718"
Sep 09 22:01:49.102586 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:01:49.102563    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718\": container with ID starting with ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718 not found: ID does not exist" containerID="ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718"
Sep 09 22:01:49.102703 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.102589    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718"} err="failed to get container status \"ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718\": rpc error: code = NotFound desc = could not find container \"ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718\": container with ID starting with ea125c8eedb82cc6dfea7df8f88434998b091640d9fe3ee0c5544e3c1745f718 not found: ID does not exist"
Sep 09 22:01:49.113548 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.113523    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"]
Sep 09 22:01:49.118924 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:49.118903    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/auth-test-model-2d3bc8-kserve-754597bd7f-bkmp4"]
Sep 09 22:01:50.998103 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:50.998070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:50.998480 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:50.998116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:51.011215 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:51.011187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:01:51.047788 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:51.047757    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b284dfd-a5c5-4199-8222-f30ee1bd62b4" path="/var/lib/kubelet/pods/5b284dfd-a5c5-4199-8222-f30ee1bd62b4/volumes"
Sep 09 22:01:51.117600 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:01:51.117572    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:02:02.402663 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.402624    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"]
Sep 09 22:02:02.403304 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.403280    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b284dfd-a5c5-4199-8222-f30ee1bd62b4" containerName="main"
Sep 09 22:02:02.403304 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.403305    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b284dfd-a5c5-4199-8222-f30ee1bd62b4" containerName="main"
Sep 09 22:02:02.403500 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.403441    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b284dfd-a5c5-4199-8222-f30ee1bd62b4" containerName="main"
Sep 09 22:02:02.408100 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.408077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.412337 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.412308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"kube-root-ca.crt\""
Sep 09 22:02:02.412337 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.412324    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"sub-test-model-64fec6-kserve-self-signed-certs\""
Sep 09 22:02:02.412337 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.412312    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"openshift-service-ca.crt\""
Sep 09 22:02:02.412740 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.412312    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-64fec6\"/\"default-dockercfg-m9z8s\""
Sep 09 22:02:02.416314 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.416284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"]
Sep 09 22:02:02.529705 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.529668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfssp\" (UniqueName: \"kubernetes.io/projected/762a8068-a55b-4c10-999e-97883f43cec6-kube-api-access-lfssp\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.529907 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.529828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762a8068-a55b-4c10-999e-97883f43cec6-tls-certs\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.529907 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.529858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-tmp-dir\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.529907 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.529885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-home\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.530017 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.529953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-model-cache\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.530017 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.529991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-dshm\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631182 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfssp\" (UniqueName: \"kubernetes.io/projected/762a8068-a55b-4c10-999e-97883f43cec6-kube-api-access-lfssp\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631335 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762a8068-a55b-4c10-999e-97883f43cec6-tls-certs\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631335 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-tmp-dir\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631335 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-home\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631335 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-model-cache\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631589 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-dshm\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631772 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-tmp-dir\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631875 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-home\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.631875 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.631804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-model-cache\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.634018 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.633991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-dshm\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.634113 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.634098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762a8068-a55b-4c10-999e-97883f43cec6-tls-certs\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.640851 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.640822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfssp\" (UniqueName: \"kubernetes.io/projected/762a8068-a55b-4c10-999e-97883f43cec6-kube-api-access-lfssp\") pod \"sub-test-model-64fec6-kserve-5887c9fb59-bk2f6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") " pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.719881 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.719779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:02.991472 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:02.991441    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"]
Sep 09 22:02:02.992957 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:02:02.992932    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod762a8068_a55b_4c10_999e_97883f43cec6.slice/crio-5327f937f8e8e531a76c7ca34e91a5778737962fcd3f4a922f8fa342191f47f4 WatchSource:0}: Error finding container 5327f937f8e8e531a76c7ca34e91a5778737962fcd3f4a922f8fa342191f47f4: Status 404 returned error can't find the container with id 5327f937f8e8e531a76c7ca34e91a5778737962fcd3f4a922f8fa342191f47f4
Sep 09 22:02:03.171329 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:03.171288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6" event={"ID":"762a8068-a55b-4c10-999e-97883f43cec6","Type":"ContainerStarted","Data":"c1e65fee96f29bc054e37524ff032d5f14d008aa1b4fab7afb332123203d7ded"}
Sep 09 22:02:03.171329 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:03.171333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6" event={"ID":"762a8068-a55b-4c10-999e-97883f43cec6","Type":"ContainerStarted","Data":"5327f937f8e8e531a76c7ca34e91a5778737962fcd3f4a922f8fa342191f47f4"}
Sep 09 22:02:03.190464 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:03.190410    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6" podStartSLOduration=1.190393762 podStartE2EDuration="1.190393762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:02:03.188154269 +0000 UTC m=+1728.674037307" watchObservedRunningTime="2026-09-09 22:02:03.190393762 +0000 UTC m=+1728.676276799"
Sep 09 22:02:09.903536 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:09.903492    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"]
Sep 09 22:02:09.906467 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:09.906442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"
Sep 09 22:02:09.909359 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:09.909334    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 22:02:09.915342 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:09.915308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"]
Sep 09 22:02:10.007569 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:10.007522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3970f4a-dfb1-4250-a178-338921ee413f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f\" (UID: \"f3970f4a-dfb1-4250-a178-338921ee413f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"
Sep 09 22:02:10.109126 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:10.109087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3970f4a-dfb1-4250-a178-338921ee413f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f\" (UID: \"f3970f4a-dfb1-4250-a178-338921ee413f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"
Sep 09 22:02:10.109778 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:10.109710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3970f4a-dfb1-4250-a178-338921ee413f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f\" (UID: \"f3970f4a-dfb1-4250-a178-338921ee413f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"
Sep 09 22:02:10.219141 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:10.219052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"
Sep 09 22:02:10.490053 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:10.490027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"]
Sep 09 22:02:10.491850 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:02:10.491827    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3970f4a_dfb1_4250_a178_338921ee413f.slice/crio-d31f64bac72acb47f6c570a6600bb7737cbfe731277f698d7dca52675351d70b WatchSource:0}: Error finding container d31f64bac72acb47f6c570a6600bb7737cbfe731277f698d7dca52675351d70b: Status 404 returned error can't find the container with id d31f64bac72acb47f6c570a6600bb7737cbfe731277f698d7dca52675351d70b
Sep 09 22:02:11.215244 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:11.215211    2563 generic.go:356] "Generic (PLEG): container finished" podID="f3970f4a-dfb1-4250-a178-338921ee413f" containerID="e43ac73f142ddb93dfecc5d8227c9a2e94621bc8d7841859554971198ff0e73b" exitCode=0
Sep 09 22:02:11.215617 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:11.215300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f" event={"ID":"f3970f4a-dfb1-4250-a178-338921ee413f","Type":"ContainerDied","Data":"e43ac73f142ddb93dfecc5d8227c9a2e94621bc8d7841859554971198ff0e73b"}
Sep 09 22:02:11.215617 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:11.215349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f" event={"ID":"f3970f4a-dfb1-4250-a178-338921ee413f","Type":"ContainerStarted","Data":"d31f64bac72acb47f6c570a6600bb7737cbfe731277f698d7dca52675351d70b"}
Sep 09 22:02:12.360335 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:12.360306    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"
Sep 09 22:02:12.535479 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:12.535392    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3970f4a-dfb1-4250-a178-338921ee413f-maas-api-service-ca\") pod \"f3970f4a-dfb1-4250-a178-338921ee413f\" (UID: \"f3970f4a-dfb1-4250-a178-338921ee413f\") "
Sep 09 22:02:12.535904 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:12.535875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3970f4a-dfb1-4250-a178-338921ee413f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3970f4a-dfb1-4250-a178-338921ee413f" (UID: "f3970f4a-dfb1-4250-a178-338921ee413f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:02:12.636260 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:12.636226    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3970f4a-dfb1-4250-a178-338921ee413f-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:12.720887 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:12.720834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:12.721095 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:12.720910    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:12.739022 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:12.738991    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:13.227599 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:13.227570    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"
Sep 09 22:02:13.227805 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:13.227602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f" event={"ID":"f3970f4a-dfb1-4250-a178-338921ee413f","Type":"ContainerDied","Data":"d31f64bac72acb47f6c570a6600bb7737cbfe731277f698d7dca52675351d70b"}
Sep 09 22:02:13.227805 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:13.227651    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d31f64bac72acb47f6c570a6600bb7737cbfe731277f698d7dca52675351d70b"
Sep 09 22:02:13.238988 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:13.238961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:02:14.755822 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:14.755790    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"]
Sep 09 22:02:14.790306 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:14.790271    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"]
Sep 09 22:02:25.432310 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:25.432270    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"]
Sep 09 22:02:25.434475 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:25.434447    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fa986-29816520-q4jn8"]
Sep 09 22:02:25.681232 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:25.681193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"]
Sep 09 22:02:25.684162 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:25.684095    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3fa986-pjp9f"]
Sep 09 22:02:27.046872 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:27.046837    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6e7cc7-cc9f-476d-8b4d-2093330dbab8" path="/var/lib/kubelet/pods/ae6e7cc7-cc9f-476d-8b4d-2093330dbab8/volumes"
Sep 09 22:02:27.047250 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:27.047160    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3970f4a-dfb1-4250-a178-338921ee413f" path="/var/lib/kubelet/pods/f3970f4a-dfb1-4250-a178-338921ee413f/volumes"
Sep 09 22:02:29.776622 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:29.776558    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp" podUID="1b8eb228-1f32-49fe-9fe8-311497507a14" containerName="main" containerID="cri-o://99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2" gracePeriod=45
Sep 09 22:02:29.810571 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:29.810537    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m" podUID="baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" containerName="main" containerID="cri-o://66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535" gracePeriod=45
Sep 09 22:02:30.088178 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.088144    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 22:02:30.091896 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.091863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 22:02:30.207675 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207641    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tls-certs\") pod \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207693    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tmp-dir\") pod \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-model-cache\") pod \"1b8eb228-1f32-49fe-9fe8-311497507a14\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-model-cache\") pod \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-dshm\") pod \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8eb228-1f32-49fe-9fe8-311497507a14-tls-certs\") pod \"1b8eb228-1f32-49fe-9fe8-311497507a14\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmlqr\" (UniqueName: \"kubernetes.io/projected/1b8eb228-1f32-49fe-9fe8-311497507a14-kube-api-access-pmlqr\") pod \"1b8eb228-1f32-49fe-9fe8-311497507a14\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207898    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-home\") pod \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgq5c\" (UniqueName: \"kubernetes.io/projected/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-kube-api-access-bgq5c\") pod \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\" (UID: \"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-home\") pod \"1b8eb228-1f32-49fe-9fe8-311497507a14\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b8eb228-1f32-49fe-9fe8-311497507a14" (UID: "1b8eb228-1f32-49fe-9fe8-311497507a14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.207998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" (UID: "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" (UID: "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-dshm\") pod \"1b8eb228-1f32-49fe-9fe8-311497507a14\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208067    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-tmp-dir\") pod \"1b8eb228-1f32-49fe-9fe8-311497507a14\" (UID: \"1b8eb228-1f32-49fe-9fe8-311497507a14\") "
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208214    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-home" (OuterVolumeSpecName: "home") pod "1b8eb228-1f32-49fe-9fe8-311497507a14" (UID: "1b8eb228-1f32-49fe-9fe8-311497507a14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.208269 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-home" (OuterVolumeSpecName: "home") pod "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" (UID: "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.209349 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208392    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b8eb228-1f32-49fe-9fe8-311497507a14" (UID: "1b8eb228-1f32-49fe-9fe8-311497507a14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.209349 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208504    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.209349 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208628    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.209349 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208644    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.209349 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208658    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.209349 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.208668    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.210229 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.210200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-dshm" (OuterVolumeSpecName: "dshm") pod "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" (UID: "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.210608 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.210589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" (UID: "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:02:30.210767 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.210745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-dshm" (OuterVolumeSpecName: "dshm") pod "1b8eb228-1f32-49fe-9fe8-311497507a14" (UID: "1b8eb228-1f32-49fe-9fe8-311497507a14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:02:30.210767 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.210743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8eb228-1f32-49fe-9fe8-311497507a14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b8eb228-1f32-49fe-9fe8-311497507a14" (UID: "1b8eb228-1f32-49fe-9fe8-311497507a14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:02:30.210767 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.210756    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-kube-api-access-bgq5c" (OuterVolumeSpecName: "kube-api-access-bgq5c") pod "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" (UID: "baf3cc34-b7c5-4bb8-80e6-c64394ca95c6"). InnerVolumeSpecName "kube-api-access-bgq5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:30.210948 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.210863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8eb228-1f32-49fe-9fe8-311497507a14-kube-api-access-pmlqr" (OuterVolumeSpecName: "kube-api-access-pmlqr") pod "1b8eb228-1f32-49fe-9fe8-311497507a14" (UID: "1b8eb228-1f32-49fe-9fe8-311497507a14"). InnerVolumeSpecName "kube-api-access-pmlqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:30.309679 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.309586    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgq5c\" (UniqueName: \"kubernetes.io/projected/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-kube-api-access-bgq5c\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.309679 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.309622    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.309679 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.309631    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8eb228-1f32-49fe-9fe8-311497507a14-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.309679 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.309642    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.309679 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.309651    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.309679 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.309660    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8eb228-1f32-49fe-9fe8-311497507a14-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.309679 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.309667    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmlqr\" (UniqueName: \"kubernetes.io/projected/1b8eb228-1f32-49fe-9fe8-311497507a14-kube-api-access-pmlqr\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:02:30.327026 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.326989    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b8eb228-1f32-49fe-9fe8-311497507a14" containerID="99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2" exitCode=0
Sep 09 22:02:30.327179 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.327037    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"
Sep 09 22:02:30.327179 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.327078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp" event={"ID":"1b8eb228-1f32-49fe-9fe8-311497507a14","Type":"ContainerDied","Data":"99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2"}
Sep 09 22:02:30.327179 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.327127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp" event={"ID":"1b8eb228-1f32-49fe-9fe8-311497507a14","Type":"ContainerDied","Data":"6dc3d77cd733e339bc6dbe1545eed035b108b118e91ac3df39e89f5924c24f31"}
Sep 09 22:02:30.327179 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.327148    2563 scope.go:117] "RemoveContainer" containerID="99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2"
Sep 09 22:02:30.328636 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.328613    2563 generic.go:356] "Generic (PLEG): container finished" podID="baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" containerID="66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535" exitCode=0
Sep 09 22:02:30.328769 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.328675    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"
Sep 09 22:02:30.328769 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.328685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m" event={"ID":"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6","Type":"ContainerDied","Data":"66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535"}
Sep 09 22:02:30.328769 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.328709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m" event={"ID":"baf3cc34-b7c5-4bb8-80e6-c64394ca95c6","Type":"ContainerDied","Data":"6d4c96e50e7a3848223e8a339e439b47cc3adbaed7663bf292a86bb3fe8f4825"}
Sep 09 22:02:30.336555 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.336534    2563 scope.go:117] "RemoveContainer" containerID="99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2"
Sep 09 22:02:30.336909 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:02:30.336884    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2\": container with ID starting with 99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2 not found: ID does not exist" containerID="99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2"
Sep 09 22:02:30.336981 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.336923    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2"} err="failed to get container status \"99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2\": rpc error: code = NotFound desc = could not find container \"99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2\": container with ID starting with 99d67d7f39bb8dbc73b08d2c5a2e9293054c2f3a28018f7ba19cc631b9447ce2 not found: ID does not exist"
Sep 09 22:02:30.336981 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.336952    2563 scope.go:117] "RemoveContainer" containerID="66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535"
Sep 09 22:02:30.346085 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.346066    2563 scope.go:117] "RemoveContainer" containerID="66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535"
Sep 09 22:02:30.346373 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:02:30.346354    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535\": container with ID starting with 66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535 not found: ID does not exist" containerID="66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535"
Sep 09 22:02:30.346411 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.346383    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535"} err="failed to get container status \"66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535\": rpc error: code = NotFound desc = could not find container \"66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535\": container with ID starting with 66ab9d35bf276cb44f681861115aef733ae4c7100e9994300e7b247d5e374535 not found: ID does not exist"
Sep 09 22:02:30.353917 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.353894    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"]
Sep 09 22:02:30.355747 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.355711    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/simulator-w4-3fa986-kserve-66f859d6c8-87m8m"]
Sep 09 22:02:30.372259 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.368122    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"]
Sep 09 22:02:30.374676 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:30.374650    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fa986/distinct-w4-3fa986-kserve-8f74d7d78-fjrbp"]
Sep 09 22:02:31.046822 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:31.046743    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8eb228-1f32-49fe-9fe8-311497507a14" path="/var/lib/kubelet/pods/1b8eb228-1f32-49fe-9fe8-311497507a14/volumes"
Sep 09 22:02:31.047180 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:02:31.047137    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" path="/var/lib/kubelet/pods/baf3cc34-b7c5-4bb8-80e6-c64394ca95c6/volumes"
Sep 09 22:03:03.117112 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:03.117069    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"]
Sep 09 22:03:04.349703 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:04.349672    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"]
Sep 09 22:03:18.137180 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.137109    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc" podUID="32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" containerName="main" containerID="cri-o://7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a" gracePeriod=45
Sep 09 22:03:18.393313 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.393255    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:03:18.469235 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tls-certs\") pod \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") "
Sep 09 22:03:18.469435 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c78c\" (UniqueName: \"kubernetes.io/projected/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-kube-api-access-2c78c\") pod \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") "
Sep 09 22:03:18.469435 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469279    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tmp-dir\") pod \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") "
Sep 09 22:03:18.469435 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469303    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-home\") pod \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") "
Sep 09 22:03:18.469435 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-dshm\") pod \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") "
Sep 09 22:03:18.469666 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469466    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-model-cache\") pod \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\" (UID: \"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f\") "
Sep 09 22:03:18.469666 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469627    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" (UID: "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:18.469825 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469672    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-home" (OuterVolumeSpecName: "home") pod "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" (UID: "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:18.469911 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.469890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" (UID: "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:18.470175 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.470113    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.470280 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.470179    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.471836 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.471803    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" (UID: "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:18.471836 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.471820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-dshm" (OuterVolumeSpecName: "dshm") pod "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" (UID: "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:18.471989 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.471876    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-kube-api-access-2c78c" (OuterVolumeSpecName: "kube-api-access-2c78c") pod "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" (UID: "32ef9aca-f1d3-463a-a12e-adc2aaa04b0f"). InnerVolumeSpecName "kube-api-access-2c78c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:18.570705 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.570668    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c78c\" (UniqueName: \"kubernetes.io/projected/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-kube-api-access-2c78c\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.570705 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.570698    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.570705 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.570710    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.570961 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.570750    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.595796 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.595754    2563 generic.go:356] "Generic (PLEG): container finished" podID="32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" containerID="7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a" exitCode=0
Sep 09 22:03:18.595962 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.595804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc" event={"ID":"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f","Type":"ContainerDied","Data":"7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a"}
Sep 09 22:03:18.595962 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.595845    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"
Sep 09 22:03:18.595962 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.595857    2563 scope.go:117] "RemoveContainer" containerID="7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a"
Sep 09 22:03:18.596152 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.595845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc" event={"ID":"32ef9aca-f1d3-463a-a12e-adc2aaa04b0f","Type":"ContainerDied","Data":"2096e6311d80dd5b420eef07a79095ff040e8589bd4d8706a90cdb0fa7a525e6"}
Sep 09 22:03:18.605934 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.605907    2563 scope.go:117] "RemoveContainer" containerID="7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a"
Sep 09 22:03:18.606216 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:03:18.606193    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a\": container with ID starting with 7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a not found: ID does not exist" containerID="7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a"
Sep 09 22:03:18.606340 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.606223    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a"} err="failed to get container status \"7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a\": rpc error: code = NotFound desc = could not find container \"7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a\": container with ID starting with 7a7c718653899896bd082f41b17faf601aaa5e8a1c83a0221413f1a11b70851a not found: ID does not exist"
Sep 09 22:03:18.617497 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.617464    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"]
Sep 09 22:03:18.623417 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:18.623389    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d3bc8/sub-test-model-2d3bc8-kserve-6dccc9cb87-hz4gc"]
Sep 09 22:03:19.046849 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.046818    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" path="/var/lib/kubelet/pods/32ef9aca-f1d3-463a-a12e-adc2aaa04b0f/volumes"
Sep 09 22:03:19.369690 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.369649    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6" podUID="762a8068-a55b-4c10-999e-97883f43cec6" containerName="main" containerID="cri-o://c1e65fee96f29bc054e37524ff032d5f14d008aa1b4fab7afb332123203d7ded" gracePeriod=45
Sep 09 22:03:19.604628 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.604591    2563 generic.go:356] "Generic (PLEG): container finished" podID="762a8068-a55b-4c10-999e-97883f43cec6" containerID="c1e65fee96f29bc054e37524ff032d5f14d008aa1b4fab7afb332123203d7ded" exitCode=0
Sep 09 22:03:19.604825 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.604653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6" event={"ID":"762a8068-a55b-4c10-999e-97883f43cec6","Type":"ContainerDied","Data":"c1e65fee96f29bc054e37524ff032d5f14d008aa1b4fab7afb332123203d7ded"}
Sep 09 22:03:19.628172 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.628103    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:03:19.779896 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.779863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-dshm\") pod \"762a8068-a55b-4c10-999e-97883f43cec6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") "
Sep 09 22:03:19.780089 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.779903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-model-cache\") pod \"762a8068-a55b-4c10-999e-97883f43cec6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") "
Sep 09 22:03:19.780089 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.779939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762a8068-a55b-4c10-999e-97883f43cec6-tls-certs\") pod \"762a8068-a55b-4c10-999e-97883f43cec6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") "
Sep 09 22:03:19.780089 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-home\") pod \"762a8068-a55b-4c10-999e-97883f43cec6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") "
Sep 09 22:03:19.780089 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780035    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-tmp-dir\") pod \"762a8068-a55b-4c10-999e-97883f43cec6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") "
Sep 09 22:03:19.780089 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780070    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfssp\" (UniqueName: \"kubernetes.io/projected/762a8068-a55b-4c10-999e-97883f43cec6-kube-api-access-lfssp\") pod \"762a8068-a55b-4c10-999e-97883f43cec6\" (UID: \"762a8068-a55b-4c10-999e-97883f43cec6\") "
Sep 09 22:03:19.780376 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-model-cache" (OuterVolumeSpecName: "model-cache") pod "762a8068-a55b-4c10-999e-97883f43cec6" (UID: "762a8068-a55b-4c10-999e-97883f43cec6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:19.780376 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-home" (OuterVolumeSpecName: "home") pod "762a8068-a55b-4c10-999e-97883f43cec6" (UID: "762a8068-a55b-4c10-999e-97883f43cec6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:19.780376 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780303    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "762a8068-a55b-4c10-999e-97883f43cec6" (UID: "762a8068-a55b-4c10-999e-97883f43cec6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:19.780509 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780389    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:19.780509 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780408    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:19.780509 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.780423    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:19.782375 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.782352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-dshm" (OuterVolumeSpecName: "dshm") pod "762a8068-a55b-4c10-999e-97883f43cec6" (UID: "762a8068-a55b-4c10-999e-97883f43cec6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:19.782459 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.782438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762a8068-a55b-4c10-999e-97883f43cec6-kube-api-access-lfssp" (OuterVolumeSpecName: "kube-api-access-lfssp") pod "762a8068-a55b-4c10-999e-97883f43cec6" (UID: "762a8068-a55b-4c10-999e-97883f43cec6"). InnerVolumeSpecName "kube-api-access-lfssp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:19.782459 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.782452    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762a8068-a55b-4c10-999e-97883f43cec6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "762a8068-a55b-4c10-999e-97883f43cec6" (UID: "762a8068-a55b-4c10-999e-97883f43cec6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:19.881324 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.881229    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762a8068-a55b-4c10-999e-97883f43cec6-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:19.881324 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.881262    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfssp\" (UniqueName: \"kubernetes.io/projected/762a8068-a55b-4c10-999e-97883f43cec6-kube-api-access-lfssp\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:19.881324 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:19.881271    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762a8068-a55b-4c10-999e-97883f43cec6-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:03:20.611151 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:20.611119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"
Sep 09 22:03:20.611646 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:20.611166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6" event={"ID":"762a8068-a55b-4c10-999e-97883f43cec6","Type":"ContainerDied","Data":"5327f937f8e8e531a76c7ca34e91a5778737962fcd3f4a922f8fa342191f47f4"}
Sep 09 22:03:20.611646 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:20.611216    2563 scope.go:117] "RemoveContainer" containerID="c1e65fee96f29bc054e37524ff032d5f14d008aa1b4fab7afb332123203d7ded"
Sep 09 22:03:20.632971 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:20.632938    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"]
Sep 09 22:03:20.637277 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:20.637251    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-64fec6/sub-test-model-64fec6-kserve-5887c9fb59-bk2f6"]
Sep 09 22:03:21.047149 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:03:21.047064    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="762a8068-a55b-4c10-999e-97883f43cec6" path="/var/lib/kubelet/pods/762a8068-a55b-4c10-999e-97883f43cec6/volumes"
Sep 09 22:05:15.430522 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:15.430486    2563 scope.go:117] "RemoveContainer" containerID="221b728f9aa3cc22e87e75230b614a2c69eac120c5c45d95ec81b4ce3ad3529c"
Sep 09 22:05:15.439485 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:15.439462    2563 scope.go:117] "RemoveContainer" containerID="e474d81d178b279ad331739912166853438bd24f34663d1646594cf714512863"
Sep 09 22:05:15.447833 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:15.447811    2563 scope.go:117] "RemoveContainer" containerID="fd8c708950950918b5364d7d924d927002802737d7df658528674e575bd43f2d"
Sep 09 22:05:37.895040 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:37.895000    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lswx7_2a06f547-5c0e-452b-8a1d-a9ab4806659c/payload-processing/0.log"
Sep 09 22:05:43.999068 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999034    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"]
Sep 09 22:05:43.999631 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999610    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8eb228-1f32-49fe-9fe8-311497507a14" containerName="main"
Sep 09 22:05:43.999631 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999632    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8eb228-1f32-49fe-9fe8-311497507a14" containerName="main"
Sep 09 22:05:43.999817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999645    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762a8068-a55b-4c10-999e-97883f43cec6" containerName="main"
Sep 09 22:05:43.999817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999653    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="762a8068-a55b-4c10-999e-97883f43cec6" containerName="main"
Sep 09 22:05:43.999817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999666    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" containerName="main"
Sep 09 22:05:43.999817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999673    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" containerName="main"
Sep 09 22:05:43.999817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999689    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" containerName="main"
Sep 09 22:05:43.999817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999697    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" containerName="main"
Sep 09 22:05:43.999817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999745    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3970f4a-dfb1-4250-a178-338921ee413f" containerName="revoke-keys"
Sep 09 22:05:43.999817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999753    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3970f4a-dfb1-4250-a178-338921ee413f" containerName="revoke-keys"
Sep 09 22:05:44.000205 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999883    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf3cc34-b7c5-4bb8-80e6-c64394ca95c6" containerName="main"
Sep 09 22:05:44.000205 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999896    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8eb228-1f32-49fe-9fe8-311497507a14" containerName="main"
Sep 09 22:05:44.000205 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999905    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="762a8068-a55b-4c10-999e-97883f43cec6" containerName="main"
Sep 09 22:05:44.000205 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999917    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ef9aca-f1d3-463a-a12e-adc2aaa04b0f" containerName="main"
Sep 09 22:05:44.000205 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:43.999926    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3970f4a-dfb1-4250-a178-338921ee413f" containerName="revoke-keys"
Sep 09 22:05:44.003222 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.003201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.007399 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.007373    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-100032\"/\"kube-root-ca.crt\""
Sep 09 22:05:44.007399 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.007390    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-100032\"/\"ipp-route-100032-kserve-self-signed-certs\""
Sep 09 22:05:44.007594 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.007391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-100032\"/\"default-dockercfg-4dn9l\""
Sep 09 22:05:44.007594 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.007390    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-100032\"/\"openshift-service-ca.crt\""
Sep 09 22:05:44.014159 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.014136    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"]
Sep 09 22:05:44.050829 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.050795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-home\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.051023 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.050854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-model-cache\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.051023 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.050974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tmp-dir\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.051023 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.051011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-dshm\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.051203 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.051097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8q5s\" (UniqueName: \"kubernetes.io/projected/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-kube-api-access-s8q5s\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.051203 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.051142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tls-certs\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.151653 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.151609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8q5s\" (UniqueName: \"kubernetes.io/projected/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-kube-api-access-s8q5s\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.151892 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.151662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tls-certs\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.151892 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.151699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-home\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.151892 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.151792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-model-cache\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.151892 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.151855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tmp-dir\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.151892 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.151883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-dshm\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.152287 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.152255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-home\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.152412 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.152268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-model-cache\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.152412 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.152310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tmp-dir\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.154229 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.154210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-dshm\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.154400 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.154384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tls-certs\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.161124 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.161103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8q5s\" (UniqueName: \"kubernetes.io/projected/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-kube-api-access-s8q5s\") pod \"ipp-route-100032-kserve-648697fddc-vhkrm\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") " pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.313998 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.313898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:44.593211 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:44.593185    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"]
Sep 09 22:05:44.595529 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:05:44.595495    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bfe8c82_c742_4f10_ad0c_35cadf9b91bd.slice/crio-40b1e861f9eb85ee5616a94f29cff885abc4914302d0feb4a23b99e277ea5db2 WatchSource:0}: Error finding container 40b1e861f9eb85ee5616a94f29cff885abc4914302d0feb4a23b99e277ea5db2: Status 404 returned error can't find the container with id 40b1e861f9eb85ee5616a94f29cff885abc4914302d0feb4a23b99e277ea5db2
Sep 09 22:05:45.393051 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:45.393009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm" event={"ID":"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd","Type":"ContainerStarted","Data":"3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e"}
Sep 09 22:05:45.393051 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:45.393058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm" event={"ID":"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd","Type":"ContainerStarted","Data":"40b1e861f9eb85ee5616a94f29cff885abc4914302d0feb4a23b99e277ea5db2"}
Sep 09 22:05:45.414812 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:45.414755    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm" podStartSLOduration=2.414736845 podStartE2EDuration="2.414736845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:45.41057803 +0000 UTC m=+1950.896461070" watchObservedRunningTime="2026-09-09 22:05:45.414736845 +0000 UTC m=+1950.900619874"
Sep 09 22:05:54.317747 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:54.317629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:54.317747 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:54.317699    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:54.330756 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:54.330710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:05:54.454008 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:05:54.453971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:06:15.474815 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:15.474695    2563 scope.go:117] "RemoveContainer" containerID="eeeb8b6399e904af55cfecb8446395a0e7232e325d3d55cb25efe8ed157d5ad9"
Sep 09 22:06:15.483408 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:15.483385    2563 scope.go:117] "RemoveContainer" containerID="b77816bfba2d5c647b0be56ba056d9be0b314e9deeeee792bde6663b5e4a0252"
Sep 09 22:06:15.491592 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:15.491564    2563 scope.go:117] "RemoveContainer" containerID="71e58621b119ce94fd8bc01b1e5ec5db9a8e45a35bc01228c636ef5df33d0a70"
Sep 09 22:06:15.499246 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:15.499224    2563 scope.go:117] "RemoveContainer" containerID="0ec2b69b4efa2f4406ad7caf9ca6779e7537ce78fba19a76a0e0b8e496c6d984"
Sep 09 22:06:24.352382 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:24.352351    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lswx7_2a06f547-5c0e-452b-8a1d-a9ab4806659c/payload-processing/0.log"
Sep 09 22:06:30.079301 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.079262    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"]
Sep 09 22:06:30.082881 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.082864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:30.087000 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.086971    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-bcb351\""
Sep 09 22:06:30.087228 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.087212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-bcb351-dockercfg-hgsgq\""
Sep 09 22:06:30.097559 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.097530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"]
Sep 09 22:06:30.099444 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.099417    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"]
Sep 09 22:06:30.102995 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.102975    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:30.111921 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.111892    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"]
Sep 09 22:06:30.197205 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.197165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ncz\" (UniqueName: \"kubernetes.io/projected/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-kube-api-access-z6ncz\") pod \"payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g\" (UID: \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:30.197375 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.197310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pv2\" (UniqueName: \"kubernetes.io/projected/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-kube-api-access-z2pv2\") pod \"payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h\" (UID: \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:30.197375 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.197363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g\" (UID: \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:30.197501 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.197439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h\" (UID: \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:30.298056 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.298012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h\" (UID: \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:30.298221 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.298082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ncz\" (UniqueName: \"kubernetes.io/projected/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-kube-api-access-z6ncz\") pod \"payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g\" (UID: \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:30.298221 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.298187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pv2\" (UniqueName: \"kubernetes.io/projected/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-kube-api-access-z2pv2\") pod \"payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h\" (UID: \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:30.298298 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.298228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g\" (UID: \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:30.299028 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.299005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h\" (UID: \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:30.299071 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.299035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g\" (UID: \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:30.308024 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.307994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ncz\" (UniqueName: \"kubernetes.io/projected/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-kube-api-access-z6ncz\") pod \"payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g\" (UID: \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:30.308248 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.308229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pv2\" (UniqueName: \"kubernetes.io/projected/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-kube-api-access-z2pv2\") pod \"payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h\" (UID: \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:30.393150 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.393117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:30.414628 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.414601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:30.669036 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.669008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"]
Sep 09 22:06:30.671061 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:06:30.671030    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fab48b_6fe3_4bf0_86e7_4ec84e203801.slice/crio-6454415d1430f21fc61c4280ee902fa0b9245dadc2fbff27dd4e79b63ffb798d WatchSource:0}: Error finding container 6454415d1430f21fc61c4280ee902fa0b9245dadc2fbff27dd4e79b63ffb798d: Status 404 returned error can't find the container with id 6454415d1430f21fc61c4280ee902fa0b9245dadc2fbff27dd4e79b63ffb798d
Sep 09 22:06:30.700896 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:30.696898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"]
Sep 09 22:06:30.704157 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:06:30.704128    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee2f778_cecf_489e_8ad2_e6bbc864f28e.slice/crio-c646e8ccf73295ce2ee16e17f8428e0eb0f86f08d84b9bf54d4e739dfe190e9e WatchSource:0}: Error finding container c646e8ccf73295ce2ee16e17f8428e0eb0f86f08d84b9bf54d4e739dfe190e9e: Status 404 returned error can't find the container with id c646e8ccf73295ce2ee16e17f8428e0eb0f86f08d84b9bf54d4e739dfe190e9e
Sep 09 22:06:31.640317 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:31.640274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g" event={"ID":"b9fab48b-6fe3-4bf0-86e7-4ec84e203801","Type":"ContainerStarted","Data":"a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe"}
Sep 09 22:06:31.640815 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:31.640346    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:31.640815 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:31.640369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g" event={"ID":"b9fab48b-6fe3-4bf0-86e7-4ec84e203801","Type":"ContainerStarted","Data":"6454415d1430f21fc61c4280ee902fa0b9245dadc2fbff27dd4e79b63ffb798d"}
Sep 09 22:06:31.642032 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:31.642002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h" event={"ID":"0ee2f778-cecf-489e-8ad2-e6bbc864f28e","Type":"ContainerStarted","Data":"ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc"}
Sep 09 22:06:31.642169 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:31.642051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:31.642169 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:31.642067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h" event={"ID":"0ee2f778-cecf-489e-8ad2-e6bbc864f28e","Type":"ContainerStarted","Data":"c646e8ccf73295ce2ee16e17f8428e0eb0f86f08d84b9bf54d4e739dfe190e9e"}
Sep 09 22:06:31.660730 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:31.660678    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g" podStartSLOduration=1.660662517 podStartE2EDuration="1.660662517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:06:31.658876748 +0000 UTC m=+1997.144759787" watchObservedRunningTime="2026-09-09 22:06:31.660662517 +0000 UTC m=+1997.146545554"
Sep 09 22:06:31.678879 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:31.678822    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h" podStartSLOduration=1.678806746 podStartE2EDuration="1.678806746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:06:31.67657982 +0000 UTC m=+1997.162462859" watchObservedRunningTime="2026-09-09 22:06:31.678806746 +0000 UTC m=+1997.164689784"
Sep 09 22:06:42.650381 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:42.650347    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:06:42.650776 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:42.650471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:06:53.799822 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:53.799782    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"]
Sep 09 22:06:53.803309 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:53.803293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"
Sep 09 22:06:53.806177 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:53.806152    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 22:06:53.810179 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:53.810155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"]
Sep 09 22:06:53.823992 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:53.823966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb735deb-1a24-457f-bb9d-f17ffe6795bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq\" (UID: \"fb735deb-1a24-457f-bb9d-f17ffe6795bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"
Sep 09 22:06:53.924538 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:53.924508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb735deb-1a24-457f-bb9d-f17ffe6795bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq\" (UID: \"fb735deb-1a24-457f-bb9d-f17ffe6795bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"
Sep 09 22:06:53.925143 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:53.925121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb735deb-1a24-457f-bb9d-f17ffe6795bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq\" (UID: \"fb735deb-1a24-457f-bb9d-f17ffe6795bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"
Sep 09 22:06:54.116352 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:54.116314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"
Sep 09 22:06:54.388340 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:54.388314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"]
Sep 09 22:06:54.389798 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:06:54.389767    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb735deb_1a24_457f_bb9d_f17ffe6795bc.slice/crio-f6af71cb8970db56a713f8c0edfb0dceb5e21a5630a971490cdb665ed65fa7eb WatchSource:0}: Error finding container f6af71cb8970db56a713f8c0edfb0dceb5e21a5630a971490cdb665ed65fa7eb: Status 404 returned error can't find the container with id f6af71cb8970db56a713f8c0edfb0dceb5e21a5630a971490cdb665ed65fa7eb
Sep 09 22:06:54.762402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:54.762301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq" event={"ID":"fb735deb-1a24-457f-bb9d-f17ffe6795bc","Type":"ContainerStarted","Data":"5ea72252a514edeeddb2693178542a7399ed96c4f20892f19d0e1aacdc4cdcaa"}
Sep 09 22:06:54.762402 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:54.762363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq" event={"ID":"fb735deb-1a24-457f-bb9d-f17ffe6795bc","Type":"ContainerStarted","Data":"f6af71cb8970db56a713f8c0edfb0dceb5e21a5630a971490cdb665ed65fa7eb"}
Sep 09 22:06:54.779089 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:54.779037    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq" podStartSLOduration=1.779022977 podStartE2EDuration="1.779022977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:06:54.777540628 +0000 UTC m=+2020.263423667" watchObservedRunningTime="2026-09-09 22:06:54.779022977 +0000 UTC m=+2020.264906015"
Sep 09 22:06:55.769613 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:55.769572    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb735deb-1a24-457f-bb9d-f17ffe6795bc" containerID="5ea72252a514edeeddb2693178542a7399ed96c4f20892f19d0e1aacdc4cdcaa" exitCode=0
Sep 09 22:06:55.770046 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:55.769659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq" event={"ID":"fb735deb-1a24-457f-bb9d-f17ffe6795bc","Type":"ContainerDied","Data":"5ea72252a514edeeddb2693178542a7399ed96c4f20892f19d0e1aacdc4cdcaa"}
Sep 09 22:06:56.907921 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:56.907893    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"
Sep 09 22:06:56.949471 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:56.949433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb735deb-1a24-457f-bb9d-f17ffe6795bc-maas-api-service-ca\") pod \"fb735deb-1a24-457f-bb9d-f17ffe6795bc\" (UID: \"fb735deb-1a24-457f-bb9d-f17ffe6795bc\") "
Sep 09 22:06:56.949806 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:56.949780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb735deb-1a24-457f-bb9d-f17ffe6795bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb735deb-1a24-457f-bb9d-f17ffe6795bc" (UID: "fb735deb-1a24-457f-bb9d-f17ffe6795bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:57.051552 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:57.051464    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb735deb-1a24-457f-bb9d-f17ffe6795bc-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:06:57.782456 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:57.782424    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"
Sep 09 22:06:57.782456 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:57.782441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq" event={"ID":"fb735deb-1a24-457f-bb9d-f17ffe6795bc","Type":"ContainerDied","Data":"f6af71cb8970db56a713f8c0edfb0dceb5e21a5630a971490cdb665ed65fa7eb"}
Sep 09 22:06:57.782670 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:57.782495    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6af71cb8970db56a713f8c0edfb0dceb5e21a5630a971490cdb665ed65fa7eb"
Sep 09 22:06:59.897688 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:59.897655    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"]
Sep 09 22:06:59.898205 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:59.897914    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h" podUID="0ee2f778-cecf-489e-8ad2-e6bbc864f28e" containerName="payload-processing" containerID="cri-o://ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc" gracePeriod=30
Sep 09 22:06:59.904098 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:59.904068    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"]
Sep 09 22:06:59.904540 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:06:59.904487    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g" podUID="b9fab48b-6fe3-4bf0-86e7-4ec84e203801" containerName="payload-pre-processing" containerID="cri-o://a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe" gracePeriod=30
Sep 09 22:07:00.123549 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.123511    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"]
Sep 09 22:07:00.130077 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.130050    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bcb351-pb5zq"]
Sep 09 22:07:00.154446 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.154388    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:07:00.180228 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.180197    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-plugins-config-volume\") pod \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\" (UID: \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\") "
Sep 09 22:07:00.180533 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.180422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2pv2\" (UniqueName: \"kubernetes.io/projected/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-kube-api-access-z2pv2\") pod \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\" (UID: \"0ee2f778-cecf-489e-8ad2-e6bbc864f28e\") "
Sep 09 22:07:00.180808 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.180767    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ee2f778-cecf-489e-8ad2-e6bbc864f28e" (UID: "0ee2f778-cecf-489e-8ad2-e6bbc864f28e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:00.182853 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.182825    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-kube-api-access-z2pv2" (OuterVolumeSpecName: "kube-api-access-z2pv2") pod "0ee2f778-cecf-489e-8ad2-e6bbc864f28e" (UID: "0ee2f778-cecf-489e-8ad2-e6bbc864f28e"). InnerVolumeSpecName "kube-api-access-z2pv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:00.196076 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.196052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:07:00.281779 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.281712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6ncz\" (UniqueName: \"kubernetes.io/projected/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-kube-api-access-z6ncz\") pod \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\" (UID: \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\") "
Sep 09 22:07:00.281779 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.281783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-plugins-config-volume\") pod \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\" (UID: \"b9fab48b-6fe3-4bf0-86e7-4ec84e203801\") "
Sep 09 22:07:00.282057 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.282042    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2pv2\" (UniqueName: \"kubernetes.io/projected/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-kube-api-access-z2pv2\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:00.282057 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.282054    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee2f778-cecf-489e-8ad2-e6bbc864f28e-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:00.282240 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.282213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9fab48b-6fe3-4bf0-86e7-4ec84e203801" (UID: "b9fab48b-6fe3-4bf0-86e7-4ec84e203801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:00.283984 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.283959    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-kube-api-access-z6ncz" (OuterVolumeSpecName: "kube-api-access-z6ncz") pod "b9fab48b-6fe3-4bf0-86e7-4ec84e203801" (UID: "b9fab48b-6fe3-4bf0-86e7-4ec84e203801"). InnerVolumeSpecName "kube-api-access-z6ncz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:00.382566 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.382531    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:00.382566 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.382560    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6ncz\" (UniqueName: \"kubernetes.io/projected/b9fab48b-6fe3-4bf0-86e7-4ec84e203801-kube-api-access-z6ncz\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:00.799499 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.799462    2563 generic.go:356] "Generic (PLEG): container finished" podID="b9fab48b-6fe3-4bf0-86e7-4ec84e203801" containerID="a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe" exitCode=0
Sep 09 22:07:00.799693 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.799510    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"
Sep 09 22:07:00.799693 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.799546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g" event={"ID":"b9fab48b-6fe3-4bf0-86e7-4ec84e203801","Type":"ContainerDied","Data":"a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe"}
Sep 09 22:07:00.799693 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.799608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g" event={"ID":"b9fab48b-6fe3-4bf0-86e7-4ec84e203801","Type":"ContainerDied","Data":"6454415d1430f21fc61c4280ee902fa0b9245dadc2fbff27dd4e79b63ffb798d"}
Sep 09 22:07:00.799693 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.799638    2563 scope.go:117] "RemoveContainer" containerID="a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe"
Sep 09 22:07:00.800975 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.800951    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ee2f778-cecf-489e-8ad2-e6bbc864f28e" containerID="ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc" exitCode=0
Sep 09 22:07:00.801081 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.801002    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"
Sep 09 22:07:00.801081 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.801008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h" event={"ID":"0ee2f778-cecf-489e-8ad2-e6bbc864f28e","Type":"ContainerDied","Data":"ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc"}
Sep 09 22:07:00.801081 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.801059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h" event={"ID":"0ee2f778-cecf-489e-8ad2-e6bbc864f28e","Type":"ContainerDied","Data":"c646e8ccf73295ce2ee16e17f8428e0eb0f86f08d84b9bf54d4e739dfe190e9e"}
Sep 09 22:07:00.810081 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.810062    2563 scope.go:117] "RemoveContainer" containerID="a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe"
Sep 09 22:07:00.810370 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:07:00.810351    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe\": container with ID starting with a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe not found: ID does not exist" containerID="a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe"
Sep 09 22:07:00.810439 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.810376    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe"} err="failed to get container status \"a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe\": rpc error: code = NotFound desc = could not find container \"a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe\": container with ID starting with a24d5d876b64d4aa53f315d79ce54a29ca9d9968137f5d56ffcf0e229e8071fe not found: ID does not exist"
Sep 09 22:07:00.810439 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.810392    2563 scope.go:117] "RemoveContainer" containerID="ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc"
Sep 09 22:07:00.818453 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.818438    2563 scope.go:117] "RemoveContainer" containerID="ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc"
Sep 09 22:07:00.818812 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:07:00.818704    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc\": container with ID starting with ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc not found: ID does not exist" containerID="ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc"
Sep 09 22:07:00.818899 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.818820    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc"} err="failed to get container status \"ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc\": rpc error: code = NotFound desc = could not find container \"ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc\": container with ID starting with ea6c07497caace6992599e5014353f376eb6cce05000f3e452ce3578c56538dc not found: ID does not exist"
Sep 09 22:07:00.825153 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.825133    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"]
Sep 09 22:07:00.828571 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.828548    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bcb351-6b677f6658-79b9g"]
Sep 09 22:07:00.839219 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.839195    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"]
Sep 09 22:07:00.842750 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:00.842728    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bcb351-7fccbcb875-dsj5h"]
Sep 09 22:07:01.046363 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:01.046329    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee2f778-cecf-489e-8ad2-e6bbc864f28e" path="/var/lib/kubelet/pods/0ee2f778-cecf-489e-8ad2-e6bbc864f28e/volumes"
Sep 09 22:07:01.046789 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:01.046774    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9fab48b-6fe3-4bf0-86e7-4ec84e203801" path="/var/lib/kubelet/pods/b9fab48b-6fe3-4bf0-86e7-4ec84e203801/volumes"
Sep 09 22:07:01.047112 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:01.047100    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb735deb-1a24-457f-bb9d-f17ffe6795bc" path="/var/lib/kubelet/pods/fb735deb-1a24-457f-bb9d-f17ffe6795bc/volumes"
Sep 09 22:07:10.107780 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.107742    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"]
Sep 09 22:07:10.108162 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108145    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee2f778-cecf-489e-8ad2-e6bbc864f28e" containerName="payload-processing"
Sep 09 22:07:10.108162 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108156    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee2f778-cecf-489e-8ad2-e6bbc864f28e" containerName="payload-processing"
Sep 09 22:07:10.108245 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108165    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb735deb-1a24-457f-bb9d-f17ffe6795bc" containerName="revoke-keys"
Sep 09 22:07:10.108245 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108171    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb735deb-1a24-457f-bb9d-f17ffe6795bc" containerName="revoke-keys"
Sep 09 22:07:10.108245 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108182    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9fab48b-6fe3-4bf0-86e7-4ec84e203801" containerName="payload-pre-processing"
Sep 09 22:07:10.108245 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108188    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9fab48b-6fe3-4bf0-86e7-4ec84e203801" containerName="payload-pre-processing"
Sep 09 22:07:10.108374 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108258    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb735deb-1a24-457f-bb9d-f17ffe6795bc" containerName="revoke-keys"
Sep 09 22:07:10.108374 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108268    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9fab48b-6fe3-4bf0-86e7-4ec84e203801" containerName="payload-pre-processing"
Sep 09 22:07:10.108374 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.108274    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee2f778-cecf-489e-8ad2-e6bbc864f28e" containerName="payload-processing"
Sep 09 22:07:10.112834 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.112818    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"
Sep 09 22:07:10.115827 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.115780    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 22:07:10.119556 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.119530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"]
Sep 09 22:07:10.179899 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.179862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b59b97-eb56-4c6e-b858-541de96d00de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm\" (UID: \"93b59b97-eb56-4c6e-b858-541de96d00de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"
Sep 09 22:07:10.281199 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.281148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b59b97-eb56-4c6e-b858-541de96d00de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm\" (UID: \"93b59b97-eb56-4c6e-b858-541de96d00de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"
Sep 09 22:07:10.281826 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.281805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b59b97-eb56-4c6e-b858-541de96d00de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm\" (UID: \"93b59b97-eb56-4c6e-b858-541de96d00de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"
Sep 09 22:07:10.425926 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.425825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"
Sep 09 22:07:10.712544 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:07:10.712512    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b59b97_eb56_4c6e_b858_541de96d00de.slice/crio-1b0bccc0f30883bf20b8ac496ba697baade1cc1de884097960d760bb8fdf0510 WatchSource:0}: Error finding container 1b0bccc0f30883bf20b8ac496ba697baade1cc1de884097960d760bb8fdf0510: Status 404 returned error can't find the container with id 1b0bccc0f30883bf20b8ac496ba697baade1cc1de884097960d760bb8fdf0510
Sep 09 22:07:10.712872 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.712706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"]
Sep 09 22:07:10.861623 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:10.861587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm" event={"ID":"93b59b97-eb56-4c6e-b858-541de96d00de","Type":"ContainerStarted","Data":"1b0bccc0f30883bf20b8ac496ba697baade1cc1de884097960d760bb8fdf0510"}
Sep 09 22:07:11.868411 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:11.868372    2563 generic.go:356] "Generic (PLEG): container finished" podID="93b59b97-eb56-4c6e-b858-541de96d00de" containerID="873a1086a8141b59c3f57a80a96529966bb7cdc5c43d89e0c6107f9f1629bc2c" exitCode=0
Sep 09 22:07:11.868877 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:11.868448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm" event={"ID":"93b59b97-eb56-4c6e-b858-541de96d00de","Type":"ContainerDied","Data":"873a1086a8141b59c3f57a80a96529966bb7cdc5c43d89e0c6107f9f1629bc2c"}
Sep 09 22:07:13.003830 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:13.003803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"
Sep 09 22:07:13.107862 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:13.107826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b59b97-eb56-4c6e-b858-541de96d00de-maas-api-service-ca\") pod \"93b59b97-eb56-4c6e-b858-541de96d00de\" (UID: \"93b59b97-eb56-4c6e-b858-541de96d00de\") "
Sep 09 22:07:13.108175 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:13.108152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b59b97-eb56-4c6e-b858-541de96d00de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93b59b97-eb56-4c6e-b858-541de96d00de" (UID: "93b59b97-eb56-4c6e-b858-541de96d00de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:13.209557 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:13.209452    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b59b97-eb56-4c6e-b858-541de96d00de-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:13.880062 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:13.880022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm" event={"ID":"93b59b97-eb56-4c6e-b858-541de96d00de","Type":"ContainerDied","Data":"1b0bccc0f30883bf20b8ac496ba697baade1cc1de884097960d760bb8fdf0510"}
Sep 09 22:07:13.880062 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:13.880040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"
Sep 09 22:07:13.880278 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:13.880076    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0bccc0f30883bf20b8ac496ba697baade1cc1de884097960d760bb8fdf0510"
Sep 09 22:07:25.967223 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:25.967187    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"]
Sep 09 22:07:25.979272 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:25.979244    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-100032-5tzvm"]
Sep 09 22:07:27.046930 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:27.046892    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b59b97-eb56-4c6e-b858-541de96d00de" path="/var/lib/kubelet/pods/93b59b97-eb56-4c6e-b858-541de96d00de/volumes"
Sep 09 22:07:31.331606 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:31.331567    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"]
Sep 09 22:07:46.352497 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.352451    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm" podUID="1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" containerName="main" containerID="cri-o://3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e" gracePeriod=45
Sep 09 22:07:46.610240 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.610173    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:07:46.704036 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8q5s\" (UniqueName: \"kubernetes.io/projected/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-kube-api-access-s8q5s\") pod \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") "
Sep 09 22:07:46.704235 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tls-certs\") pod \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") "
Sep 09 22:07:46.704235 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-model-cache\") pod \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") "
Sep 09 22:07:46.704235 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704152    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tmp-dir\") pod \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") "
Sep 09 22:07:46.704235 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-home\") pod \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") "
Sep 09 22:07:46.704235 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-dshm\") pod \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\" (UID: \"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd\") "
Sep 09 22:07:46.704520 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704443    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" (UID: "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:46.704520 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" (UID: "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:46.704520 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704464    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-home" (OuterVolumeSpecName: "home") pod "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" (UID: "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:46.704688 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704562    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:46.704688 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704590    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:46.704688 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.704598    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:46.706514 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.706481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-dshm" (OuterVolumeSpecName: "dshm") pod "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" (UID: "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:46.706514 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.706481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-kube-api-access-s8q5s" (OuterVolumeSpecName: "kube-api-access-s8q5s") pod "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" (UID: "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd"). InnerVolumeSpecName "kube-api-access-s8q5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:46.706668 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.706539    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" (UID: "1bfe8c82-c742-4f10-ad0c-35cadf9b91bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:07:46.806238 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.806201    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:46.806238 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.806239    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8q5s\" (UniqueName: \"kubernetes.io/projected/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-kube-api-access-s8q5s\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:46.806451 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:46.806254    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:07:47.055877 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.055845    2563 generic.go:356] "Generic (PLEG): container finished" podID="1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" containerID="3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e" exitCode=0
Sep 09 22:07:47.055999 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.055884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"
Sep 09 22:07:47.055999 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.055921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm" event={"ID":"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd","Type":"ContainerDied","Data":"3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e"}
Sep 09 22:07:47.055999 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.055966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm" event={"ID":"1bfe8c82-c742-4f10-ad0c-35cadf9b91bd","Type":"ContainerDied","Data":"40b1e861f9eb85ee5616a94f29cff885abc4914302d0feb4a23b99e277ea5db2"}
Sep 09 22:07:47.055999 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.055984    2563 scope.go:117] "RemoveContainer" containerID="3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e"
Sep 09 22:07:47.064631 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.064611    2563 scope.go:117] "RemoveContainer" containerID="3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e"
Sep 09 22:07:47.064912 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:07:47.064895    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e\": container with ID starting with 3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e not found: ID does not exist" containerID="3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e"
Sep 09 22:07:47.064970 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.064920    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e"} err="failed to get container status \"3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e\": rpc error: code = NotFound desc = could not find container \"3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e\": container with ID starting with 3ec82deae972056a403766d77b6c585fcb4f5d85ab4f5bde6963f0dff4c0365e not found: ID does not exist"
Sep 09 22:07:47.077050 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.077020    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"]
Sep 09 22:07:47.080502 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:47.080480    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-100032/ipp-route-100032-kserve-648697fddc-vhkrm"]
Sep 09 22:07:49.048157 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:07:49.048123    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" path="/var/lib/kubelet/pods/1bfe8c82-c742-4f10-ad0c-35cadf9b91bd/volumes"
Sep 09 22:08:15.547992 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:15.547955    2563 scope.go:117] "RemoveContainer" containerID="e43ac73f142ddb93dfecc5d8227c9a2e94621bc8d7841859554971198ff0e73b"
Sep 09 22:08:29.669949 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.669909    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"]
Sep 09 22:08:29.670340 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.670310    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" containerName="main"
Sep 09 22:08:29.670340 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.670321    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" containerName="main"
Sep 09 22:08:29.670340 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.670330    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b59b97-eb56-4c6e-b858-541de96d00de" containerName="revoke-keys"
Sep 09 22:08:29.670340 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.670335    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b59b97-eb56-4c6e-b858-541de96d00de" containerName="revoke-keys"
Sep 09 22:08:29.670476 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.670395    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bfe8c82-c742-4f10-ad0c-35cadf9b91bd" containerName="main"
Sep 09 22:08:29.670476 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.670404    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b59b97-eb56-4c6e-b858-541de96d00de" containerName="revoke-keys"
Sep 09 22:08:29.673484 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.673468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"
Sep 09 22:08:29.676357 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.676332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 22:08:29.682179 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.682154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"]
Sep 09 22:08:29.707133 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.707103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dc5315-894f-462e-936f-5eab755533b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2\" (UID: \"05dc5315-894f-462e-936f-5eab755533b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"
Sep 09 22:08:29.808193 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.808148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dc5315-894f-462e-936f-5eab755533b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2\" (UID: \"05dc5315-894f-462e-936f-5eab755533b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"
Sep 09 22:08:29.808823 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.808802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dc5315-894f-462e-936f-5eab755533b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2\" (UID: \"05dc5315-894f-462e-936f-5eab755533b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"
Sep 09 22:08:29.985124 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:29.985040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"
Sep 09 22:08:30.254995 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:30.254971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"]
Sep 09 22:08:30.257059 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:08:30.257032    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dc5315_894f_462e_936f_5eab755533b7.slice/crio-b7d09ecdea8558bc8902cbdd808a84f4071ab0d808845173d1c061eaf1269a6d WatchSource:0}: Error finding container b7d09ecdea8558bc8902cbdd808a84f4071ab0d808845173d1c061eaf1269a6d: Status 404 returned error can't find the container with id b7d09ecdea8558bc8902cbdd808a84f4071ab0d808845173d1c061eaf1269a6d
Sep 09 22:08:30.281149 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:30.281121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2" event={"ID":"05dc5315-894f-462e-936f-5eab755533b7","Type":"ContainerStarted","Data":"b7d09ecdea8558bc8902cbdd808a84f4071ab0d808845173d1c061eaf1269a6d"}
Sep 09 22:08:31.287691 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:31.287599    2563 generic.go:356] "Generic (PLEG): container finished" podID="05dc5315-894f-462e-936f-5eab755533b7" containerID="4ccde2bfa9a02477e7abc73bed6eff8eb6be774e3c5cfad28d51853069c98fc1" exitCode=0
Sep 09 22:08:31.287691 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:31.287667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2" event={"ID":"05dc5315-894f-462e-936f-5eab755533b7","Type":"ContainerDied","Data":"4ccde2bfa9a02477e7abc73bed6eff8eb6be774e3c5cfad28d51853069c98fc1"}
Sep 09 22:08:32.426937 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:32.426905    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"
Sep 09 22:08:32.531554 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:32.531518    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dc5315-894f-462e-936f-5eab755533b7-maas-api-service-ca\") pod \"05dc5315-894f-462e-936f-5eab755533b7\" (UID: \"05dc5315-894f-462e-936f-5eab755533b7\") "
Sep 09 22:08:32.531940 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:32.531917    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05dc5315-894f-462e-936f-5eab755533b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05dc5315-894f-462e-936f-5eab755533b7" (UID: "05dc5315-894f-462e-936f-5eab755533b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:08:32.633082 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:32.633047    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dc5315-894f-462e-936f-5eab755533b7-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:08:33.298455 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:33.298419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2" event={"ID":"05dc5315-894f-462e-936f-5eab755533b7","Type":"ContainerDied","Data":"b7d09ecdea8558bc8902cbdd808a84f4071ab0d808845173d1c061eaf1269a6d"}
Sep 09 22:08:33.298620 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:33.298473    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7d09ecdea8558bc8902cbdd808a84f4071ab0d808845173d1c061eaf1269a6d"
Sep 09 22:08:33.298620 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:33.298433    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"
Sep 09 22:08:40.506194 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:40.506158    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"]
Sep 09 22:08:40.513877 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:40.513845    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-855e86-2qjx2"]
Sep 09 22:08:41.045897 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:41.045861    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05dc5315-894f-462e-936f-5eab755533b7" path="/var/lib/kubelet/pods/05dc5315-894f-462e-936f-5eab755533b7/volumes"
Sep 09 22:08:47.887285 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:47.887247    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"]
Sep 09 22:08:47.887817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:47.887699    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05dc5315-894f-462e-936f-5eab755533b7" containerName="revoke-keys"
Sep 09 22:08:47.887817 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:47.887732    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05dc5315-894f-462e-936f-5eab755533b7" containerName="revoke-keys"
Sep 09 22:08:47.887898 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:47.887872    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05dc5315-894f-462e-936f-5eab755533b7" containerName="revoke-keys"
Sep 09 22:08:47.891022 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:47.891006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"
Sep 09 22:08:47.893911 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:47.893892    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 22:08:47.898975 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:47.898947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"]
Sep 09 22:08:47.975512 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:47.975468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee8446-a710-4fb6-9762-f274c34aef65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd\" (UID: \"fdee8446-a710-4fb6-9762-f274c34aef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"
Sep 09 22:08:48.076680 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:48.076649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee8446-a710-4fb6-9762-f274c34aef65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd\" (UID: \"fdee8446-a710-4fb6-9762-f274c34aef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"
Sep 09 22:08:48.077337 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:48.077316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee8446-a710-4fb6-9762-f274c34aef65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd\" (UID: \"fdee8446-a710-4fb6-9762-f274c34aef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"
Sep 09 22:08:48.203367 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:48.203259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"
Sep 09 22:08:48.475634 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:48.475610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"]
Sep 09 22:08:48.477646 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:08:48.477622    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdee8446_a710_4fb6_9762_f274c34aef65.slice/crio-87c9f6513066e5b97589d14a31a467cdf8ca842620fb8a924cc87a826bdfae85 WatchSource:0}: Error finding container 87c9f6513066e5b97589d14a31a467cdf8ca842620fb8a924cc87a826bdfae85: Status 404 returned error can't find the container with id 87c9f6513066e5b97589d14a31a467cdf8ca842620fb8a924cc87a826bdfae85
Sep 09 22:08:49.386256 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:49.386223    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdee8446-a710-4fb6-9762-f274c34aef65" containerID="6866ce6897b7f2faa6764e9f4098a670ac01bb23ca179172eb1d97daf4e9729a" exitCode=0
Sep 09 22:08:49.386649 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:49.386275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd" event={"ID":"fdee8446-a710-4fb6-9762-f274c34aef65","Type":"ContainerDied","Data":"6866ce6897b7f2faa6764e9f4098a670ac01bb23ca179172eb1d97daf4e9729a"}
Sep 09 22:08:49.386649 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:49.386317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd" event={"ID":"fdee8446-a710-4fb6-9762-f274c34aef65","Type":"ContainerStarted","Data":"87c9f6513066e5b97589d14a31a467cdf8ca842620fb8a924cc87a826bdfae85"}
Sep 09 22:08:50.526483 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:50.526457    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"
Sep 09 22:08:50.605199 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:50.605161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee8446-a710-4fb6-9762-f274c34aef65-maas-api-service-ca\") pod \"fdee8446-a710-4fb6-9762-f274c34aef65\" (UID: \"fdee8446-a710-4fb6-9762-f274c34aef65\") "
Sep 09 22:08:50.605541 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:50.605516    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdee8446-a710-4fb6-9762-f274c34aef65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdee8446-a710-4fb6-9762-f274c34aef65" (UID: "fdee8446-a710-4fb6-9762-f274c34aef65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:08:50.706917 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:50.706831    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee8446-a710-4fb6-9762-f274c34aef65-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:08:51.398173 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:51.398145    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"
Sep 09 22:08:51.398344 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:51.398173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd" event={"ID":"fdee8446-a710-4fb6-9762-f274c34aef65","Type":"ContainerDied","Data":"87c9f6513066e5b97589d14a31a467cdf8ca842620fb8a924cc87a826bdfae85"}
Sep 09 22:08:51.398344 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:51.398218    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87c9f6513066e5b97589d14a31a467cdf8ca842620fb8a924cc87a826bdfae85"
Sep 09 22:08:58.407014 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.406973    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"]
Sep 09 22:08:58.409796 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.407282    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb" podUID="3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961" containerName="payload-processing" containerID="cri-o://03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356" gracePeriod=30
Sep 09 22:08:58.416815 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.416785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"]
Sep 09 22:08:58.417193 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.417055    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6" podUID="64335646-4cbc-4eaf-a4f2-3687fdea34fb" containerName="payload-pre-processing" containerID="cri-o://64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e" gracePeriod=30
Sep 09 22:08:58.593826 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.593765    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"]
Sep 09 22:08:58.597994 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.597969    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d3bc8-x9khd"]
Sep 09 22:08:58.690258 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.690236    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 22:08:58.692976 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.692954    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 22:08:58.784992 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.784956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64335646-4cbc-4eaf-a4f2-3687fdea34fb-plugins-config-volume\") pod \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\" (UID: \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\") "
Sep 09 22:08:58.785185 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.785053    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb9db\" (UniqueName: \"kubernetes.io/projected/64335646-4cbc-4eaf-a4f2-3687fdea34fb-kube-api-access-jb9db\") pod \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\" (UID: \"64335646-4cbc-4eaf-a4f2-3687fdea34fb\") "
Sep 09 22:08:58.785185 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.785106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-plugins-config-volume\") pod \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\" (UID: \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\") "
Sep 09 22:08:58.785185 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.785130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx5rf\" (UniqueName: \"kubernetes.io/projected/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-kube-api-access-bx5rf\") pod \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\" (UID: \"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961\") "
Sep 09 22:08:58.785383 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.785354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64335646-4cbc-4eaf-a4f2-3687fdea34fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64335646-4cbc-4eaf-a4f2-3687fdea34fb" (UID: "64335646-4cbc-4eaf-a4f2-3687fdea34fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:08:58.785483 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.785459    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961" (UID: "3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:08:58.787389 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.787363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64335646-4cbc-4eaf-a4f2-3687fdea34fb-kube-api-access-jb9db" (OuterVolumeSpecName: "kube-api-access-jb9db") pod "64335646-4cbc-4eaf-a4f2-3687fdea34fb" (UID: "64335646-4cbc-4eaf-a4f2-3687fdea34fb"). InnerVolumeSpecName "kube-api-access-jb9db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:08:58.787470 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.787411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-kube-api-access-bx5rf" (OuterVolumeSpecName: "kube-api-access-bx5rf") pod "3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961" (UID: "3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961"). InnerVolumeSpecName "kube-api-access-bx5rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:08:58.886197 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.886160    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64335646-4cbc-4eaf-a4f2-3687fdea34fb-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:08:58.886197 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.886188    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb9db\" (UniqueName: \"kubernetes.io/projected/64335646-4cbc-4eaf-a4f2-3687fdea34fb-kube-api-access-jb9db\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:08:58.886197 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.886198    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:08:58.886444 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:58.886206    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5rf\" (UniqueName: \"kubernetes.io/projected/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961-kube-api-access-bx5rf\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:08:59.046647 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.046564    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdee8446-a710-4fb6-9762-f274c34aef65" path="/var/lib/kubelet/pods/fdee8446-a710-4fb6-9762-f274c34aef65/volumes"
Sep 09 22:08:59.446136 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.446102    2563 generic.go:356] "Generic (PLEG): container finished" podID="64335646-4cbc-4eaf-a4f2-3687fdea34fb" containerID="64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e" exitCode=0
Sep 09 22:08:59.446550 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.446147    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"
Sep 09 22:08:59.446550 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.446195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6" event={"ID":"64335646-4cbc-4eaf-a4f2-3687fdea34fb","Type":"ContainerDied","Data":"64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e"}
Sep 09 22:08:59.446550 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.446246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6" event={"ID":"64335646-4cbc-4eaf-a4f2-3687fdea34fb","Type":"ContainerDied","Data":"52be59b791dd4ce79bd713fbeefdb1ac06c9f977bf8edb8a3355d5b005433386"}
Sep 09 22:08:59.446550 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.446268    2563 scope.go:117] "RemoveContainer" containerID="64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e"
Sep 09 22:08:59.447648 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.447628    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961" containerID="03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356" exitCode=0
Sep 09 22:08:59.447746 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.447676    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"
Sep 09 22:08:59.447746 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.447679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb" event={"ID":"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961","Type":"ContainerDied","Data":"03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356"}
Sep 09 22:08:59.447856 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.447779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb" event={"ID":"3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961","Type":"ContainerDied","Data":"8f6e579ccd9f1396e72864d5f6e53381cb3ceba9e3fcf8e8eacf311f530cf118"}
Sep 09 22:08:59.455447 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.455431    2563 scope.go:117] "RemoveContainer" containerID="64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e"
Sep 09 22:08:59.455685 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:08:59.455663    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e\": container with ID starting with 64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e not found: ID does not exist" containerID="64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e"
Sep 09 22:08:59.455756 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.455693    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e"} err="failed to get container status \"64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e\": rpc error: code = NotFound desc = could not find container \"64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e\": container with ID starting with 64c90e394228d65f5e6d2a138b148608618f6578e94e4c3ee15419d6bbdd377e not found: ID does not exist"
Sep 09 22:08:59.455756 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.455709    2563 scope.go:117] "RemoveContainer" containerID="03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356"
Sep 09 22:08:59.463560 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.463543    2563 scope.go:117] "RemoveContainer" containerID="03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356"
Sep 09 22:08:59.463905 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:08:59.463886    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356\": container with ID starting with 03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356 not found: ID does not exist" containerID="03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356"
Sep 09 22:08:59.463955 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.463914    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356"} err="failed to get container status \"03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356\": rpc error: code = NotFound desc = could not find container \"03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356\": container with ID starting with 03df09037d6b4710d92f82b3b88d3a8aff42f13d85fea1dad1a41b0be35b8356 not found: ID does not exist"
Sep 09 22:08:59.468408 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.468387    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"]
Sep 09 22:08:59.472604 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.472584    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d3bc8-54b8c58546-l2vl6"]
Sep 09 22:08:59.490997 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.490966    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"]
Sep 09 22:08:59.494493 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:08:59.494471    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d3bc8-c768b769f-8t5xb"]
Sep 09 22:09:01.046922 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:01.046887    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961" path="/var/lib/kubelet/pods/3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961/volumes"
Sep 09 22:09:01.047306 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:01.047250    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64335646-4cbc-4eaf-a4f2-3687fdea34fb" path="/var/lib/kubelet/pods/64335646-4cbc-4eaf-a4f2-3687fdea34fb/volumes"
Sep 09 22:09:16.628755 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.628698    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"]
Sep 09 22:09:16.633541 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.633516    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-64fec6-29816520-v5b75"]
Sep 09 22:09:16.666823 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.666785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"]
Sep 09 22:09:16.667100 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.667045    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94" podUID="549b94d9-1d2d-4b3b-9311-e961e8232c83" containerName="payload-processing" containerID="cri-o://61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8" gracePeriod=30
Sep 09 22:09:16.676733 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.676690    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"]
Sep 09 22:09:16.676982 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.676929    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64" podUID="5498223b-6218-4ff5-ae2b-aa4c95432465" containerName="payload-pre-processing" containerID="cri-o://04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0" gracePeriod=30
Sep 09 22:09:16.922312 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.922287    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 22:09:16.950216 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.950189    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549b94d9-1d2d-4b3b-9311-e961e8232c83-plugins-config-volume\") pod \"549b94d9-1d2d-4b3b-9311-e961e8232c83\" (UID: \"549b94d9-1d2d-4b3b-9311-e961e8232c83\") "
Sep 09 22:09:16.950375 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.950270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9z55\" (UniqueName: \"kubernetes.io/projected/549b94d9-1d2d-4b3b-9311-e961e8232c83-kube-api-access-j9z55\") pod \"549b94d9-1d2d-4b3b-9311-e961e8232c83\" (UID: \"549b94d9-1d2d-4b3b-9311-e961e8232c83\") "
Sep 09 22:09:16.950569 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.950546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549b94d9-1d2d-4b3b-9311-e961e8232c83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "549b94d9-1d2d-4b3b-9311-e961e8232c83" (UID: "549b94d9-1d2d-4b3b-9311-e961e8232c83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:16.952452 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.952425    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549b94d9-1d2d-4b3b-9311-e961e8232c83-kube-api-access-j9z55" (OuterVolumeSpecName: "kube-api-access-j9z55") pod "549b94d9-1d2d-4b3b-9311-e961e8232c83" (UID: "549b94d9-1d2d-4b3b-9311-e961e8232c83"). InnerVolumeSpecName "kube-api-access-j9z55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:16.952794 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:16.952772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 22:09:17.046289 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.046260    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84efaf3-e314-4e70-bb30-a138d3b2f021" path="/var/lib/kubelet/pods/d84efaf3-e314-4e70-bb30-a138d3b2f021/volumes"
Sep 09 22:09:17.051268 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.051241    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5498223b-6218-4ff5-ae2b-aa4c95432465-plugins-config-volume\") pod \"5498223b-6218-4ff5-ae2b-aa4c95432465\" (UID: \"5498223b-6218-4ff5-ae2b-aa4c95432465\") "
Sep 09 22:09:17.051416 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.051285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pfmn\" (UniqueName: \"kubernetes.io/projected/5498223b-6218-4ff5-ae2b-aa4c95432465-kube-api-access-9pfmn\") pod \"5498223b-6218-4ff5-ae2b-aa4c95432465\" (UID: \"5498223b-6218-4ff5-ae2b-aa4c95432465\") "
Sep 09 22:09:17.051482 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.051441    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549b94d9-1d2d-4b3b-9311-e961e8232c83-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:09:17.051482 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.051452    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9z55\" (UniqueName: \"kubernetes.io/projected/549b94d9-1d2d-4b3b-9311-e961e8232c83-kube-api-access-j9z55\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:09:17.051598 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.051568    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5498223b-6218-4ff5-ae2b-aa4c95432465-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5498223b-6218-4ff5-ae2b-aa4c95432465" (UID: "5498223b-6218-4ff5-ae2b-aa4c95432465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:17.053470 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.053445    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5498223b-6218-4ff5-ae2b-aa4c95432465-kube-api-access-9pfmn" (OuterVolumeSpecName: "kube-api-access-9pfmn") pod "5498223b-6218-4ff5-ae2b-aa4c95432465" (UID: "5498223b-6218-4ff5-ae2b-aa4c95432465"). InnerVolumeSpecName "kube-api-access-9pfmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:17.151995 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.151912    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5498223b-6218-4ff5-ae2b-aa4c95432465-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:09:17.151995 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.151938    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfmn\" (UniqueName: \"kubernetes.io/projected/5498223b-6218-4ff5-ae2b-aa4c95432465-kube-api-access-9pfmn\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:09:17.538491 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.538386    2563 generic.go:356] "Generic (PLEG): container finished" podID="549b94d9-1d2d-4b3b-9311-e961e8232c83" containerID="61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8" exitCode=0
Sep 09 22:09:17.538491 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.538436    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"
Sep 09 22:09:17.538491 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.538476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94" event={"ID":"549b94d9-1d2d-4b3b-9311-e961e8232c83","Type":"ContainerDied","Data":"61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8"}
Sep 09 22:09:17.538815 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.538524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94" event={"ID":"549b94d9-1d2d-4b3b-9311-e961e8232c83","Type":"ContainerDied","Data":"0eda736f789144244aee5e22a8de8f9c55699c7e985c1a97961dfcb80d27b48c"}
Sep 09 22:09:17.538815 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.538546    2563 scope.go:117] "RemoveContainer" containerID="61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8"
Sep 09 22:09:17.540006 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.539982    2563 generic.go:356] "Generic (PLEG): container finished" podID="5498223b-6218-4ff5-ae2b-aa4c95432465" containerID="04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0" exitCode=0
Sep 09 22:09:17.540126 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.540029    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"
Sep 09 22:09:17.540126 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.540034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64" event={"ID":"5498223b-6218-4ff5-ae2b-aa4c95432465","Type":"ContainerDied","Data":"04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0"}
Sep 09 22:09:17.540126 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.540083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64" event={"ID":"5498223b-6218-4ff5-ae2b-aa4c95432465","Type":"ContainerDied","Data":"09fbc741513393afc7b4eb7d8d1954ad3c094b53486a8440df040e9f0ead626f"}
Sep 09 22:09:17.550553 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.550530    2563 scope.go:117] "RemoveContainer" containerID="61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8"
Sep 09 22:09:17.550940 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:09:17.550919    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8\": container with ID starting with 61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8 not found: ID does not exist" containerID="61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8"
Sep 09 22:09:17.551047 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.550946    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8"} err="failed to get container status \"61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8\": rpc error: code = NotFound desc = could not find container \"61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8\": container with ID starting with 61fb64f4be2dcbbfa7d2e670acbe427a68b2dd2a7ea0b1313668b35d7db42af8 not found: ID does not exist"
Sep 09 22:09:17.551047 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.550964    2563 scope.go:117] "RemoveContainer" containerID="04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0"
Sep 09 22:09:17.559450 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.559412    2563 scope.go:117] "RemoveContainer" containerID="04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0"
Sep 09 22:09:17.559791 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:09:17.559772    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0\": container with ID starting with 04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0 not found: ID does not exist" containerID="04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0"
Sep 09 22:09:17.559849 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.559799    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0"} err="failed to get container status \"04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0\": rpc error: code = NotFound desc = could not find container \"04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0\": container with ID starting with 04d76d91733f6a4280caef55f11ec55785b52e586decf78608c4786a06226fd0 not found: ID does not exist"
Sep 09 22:09:17.560543 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.560525    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"]
Sep 09 22:09:17.564216 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.564197    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-64fec6-866767cccc-szm94"]
Sep 09 22:09:17.575999 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.575975    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"]
Sep 09 22:09:17.582410 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:17.582389    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-64fec6-84dff8b554-kpb64"]
Sep 09 22:09:19.046278 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:19.046232    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5498223b-6218-4ff5-ae2b-aa4c95432465" path="/var/lib/kubelet/pods/5498223b-6218-4ff5-ae2b-aa4c95432465/volumes"
Sep 09 22:09:19.046992 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:09:19.046972    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549b94d9-1d2d-4b3b-9311-e961e8232c83" path="/var/lib/kubelet/pods/549b94d9-1d2d-4b3b-9311-e961e8232c83/volumes"
Sep 09 22:10:15.589914 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:10:15.589882    2563 scope.go:117] "RemoveContainer" containerID="165a36dec798ada525fba358ff09ed048285071417d7d2e9751c6045f44fbab1"
Sep 09 22:13:15.634701 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:15.634666    2563 scope.go:117] "RemoveContainer" containerID="873a1086a8141b59c3f57a80a96529966bb7cdc5c43d89e0c6107f9f1629bc2c"
Sep 09 22:13:15.644844 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:15.644818    2563 scope.go:117] "RemoveContainer" containerID="5ea72252a514edeeddb2693178542a7399ed96c4f20892f19d0e1aacdc4cdcaa"
Sep 09 22:13:37.698298 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698214    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"]
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698604    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64335646-4cbc-4eaf-a4f2-3687fdea34fb" containerName="payload-pre-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698617    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="64335646-4cbc-4eaf-a4f2-3687fdea34fb" containerName="payload-pre-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698628    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdee8446-a710-4fb6-9762-f274c34aef65" containerName="revoke-keys"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698633    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdee8446-a710-4fb6-9762-f274c34aef65" containerName="revoke-keys"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698649    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5498223b-6218-4ff5-ae2b-aa4c95432465" containerName="payload-pre-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698655    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5498223b-6218-4ff5-ae2b-aa4c95432465" containerName="payload-pre-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698666    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961" containerName="payload-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698671    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961" containerName="payload-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698676    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549b94d9-1d2d-4b3b-9311-e961e8232c83" containerName="payload-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698681    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="549b94d9-1d2d-4b3b-9311-e961e8232c83" containerName="payload-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698759    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdee8446-a710-4fb6-9762-f274c34aef65" containerName="revoke-keys"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698768    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1c4bbc-710f-4e78-a31f-d1fbdc9ce961" containerName="payload-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698779    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="64335646-4cbc-4eaf-a4f2-3687fdea34fb" containerName="payload-pre-processing"
Sep 09 22:13:37.698781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698786    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5498223b-6218-4ff5-ae2b-aa4c95432465" containerName="payload-pre-processing"
Sep 09 22:13:37.699239 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.698794    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="549b94d9-1d2d-4b3b-9311-e961e8232c83" containerName="payload-processing"
Sep 09 22:13:37.701937 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.701921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.705011 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.704990    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fcc30655-kserve-self-signed-certs\""
Sep 09 22:13:37.714072 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.714043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"]
Sep 09 22:13:37.783330 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.783295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-dshm\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.783330 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.783333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-model-cache\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.783543 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.783359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8811c72-9a1f-40df-9b43-3beeead27c53-tls-certs\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.783543 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.783418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-home\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.783543 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.783468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47m8n\" (UniqueName: \"kubernetes.io/projected/e8811c72-9a1f-40df-9b43-3beeead27c53-kube-api-access-47m8n\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.783543 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.783489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-tmp-dir\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.884799 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.884757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-home\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.885015 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.884839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47m8n\" (UniqueName: \"kubernetes.io/projected/e8811c72-9a1f-40df-9b43-3beeead27c53-kube-api-access-47m8n\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.885015 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.884863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-tmp-dir\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.885149 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.885035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-dshm\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.885149 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.885075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-model-cache\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.885149 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.885120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8811c72-9a1f-40df-9b43-3beeead27c53-tls-certs\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.885294 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.885227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-home\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.885294 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.885238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-tmp-dir\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.885476 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.885459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-model-cache\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.887532 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.887502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-dshm\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.887781 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.887763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8811c72-9a1f-40df-9b43-3beeead27c53-tls-certs\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:37.893197 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:37.893178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47m8n\" (UniqueName: \"kubernetes.io/projected/e8811c72-9a1f-40df-9b43-3beeead27c53-kube-api-access-47m8n\") pod \"e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") " pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:38.011170 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:38.011075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:38.284944 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:38.284914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"]
Sep 09 22:13:38.287039 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:13:38.287004    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8811c72_9a1f_40df_9b43_3beeead27c53.slice/crio-6161d900904a783e1defa3a7b5f9d0eed978bff0f0bd5fa5c1f1a21b79be7135 WatchSource:0}: Error finding container 6161d900904a783e1defa3a7b5f9d0eed978bff0f0bd5fa5c1f1a21b79be7135: Status 404 returned error can't find the container with id 6161d900904a783e1defa3a7b5f9d0eed978bff0f0bd5fa5c1f1a21b79be7135
Sep 09 22:13:38.834382 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:38.834346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54" event={"ID":"e8811c72-9a1f-40df-9b43-3beeead27c53","Type":"ContainerStarted","Data":"6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4"}
Sep 09 22:13:38.834788 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:38.834394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54" event={"ID":"e8811c72-9a1f-40df-9b43-3beeead27c53","Type":"ContainerStarted","Data":"6161d900904a783e1defa3a7b5f9d0eed978bff0f0bd5fa5c1f1a21b79be7135"}
Sep 09 22:13:38.852403 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:38.852346    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54" podStartSLOduration=1.8523297730000001 podStartE2EDuration="1.852329773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:13:38.85048773 +0000 UTC m=+2424.336370770" watchObservedRunningTime="2026-09-09 22:13:38.852329773 +0000 UTC m=+2424.338212811"
Sep 09 22:13:48.011461 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:48.011428    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:48.011941 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:48.011487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:48.024878 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:48.024845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:13:48.896239 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:13:48.896209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:14:31.212531 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:31.212467    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"]
Sep 09 22:14:46.233657 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.233609    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54" podUID="e8811c72-9a1f-40df-9b43-3beeead27c53" containerName="main" containerID="cri-o://6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4" gracePeriod=45
Sep 09 22:14:46.476807 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.476781    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:14:46.539356 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539263    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-tmp-dir\") pod \"e8811c72-9a1f-40df-9b43-3beeead27c53\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") "
Sep 09 22:14:46.539356 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47m8n\" (UniqueName: \"kubernetes.io/projected/e8811c72-9a1f-40df-9b43-3beeead27c53-kube-api-access-47m8n\") pod \"e8811c72-9a1f-40df-9b43-3beeead27c53\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") "
Sep 09 22:14:46.539594 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-dshm\") pod \"e8811c72-9a1f-40df-9b43-3beeead27c53\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") "
Sep 09 22:14:46.539594 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-model-cache\") pod \"e8811c72-9a1f-40df-9b43-3beeead27c53\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") "
Sep 09 22:14:46.539594 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539430    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-home\") pod \"e8811c72-9a1f-40df-9b43-3beeead27c53\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") "
Sep 09 22:14:46.539594 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8811c72-9a1f-40df-9b43-3beeead27c53-tls-certs\") pod \"e8811c72-9a1f-40df-9b43-3beeead27c53\" (UID: \"e8811c72-9a1f-40df-9b43-3beeead27c53\") "
Sep 09 22:14:46.539594 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539559    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8811c72-9a1f-40df-9b43-3beeead27c53" (UID: "e8811c72-9a1f-40df-9b43-3beeead27c53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:14:46.539888 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539667    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-home" (OuterVolumeSpecName: "home") pod "e8811c72-9a1f-40df-9b43-3beeead27c53" (UID: "e8811c72-9a1f-40df-9b43-3beeead27c53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:14:46.539888 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539711    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8811c72-9a1f-40df-9b43-3beeead27c53" (UID: "e8811c72-9a1f-40df-9b43-3beeead27c53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:14:46.539995 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539923    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-model-cache\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:14:46.539995 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539939    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-home\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:14:46.539995 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.539948    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-tmp-dir\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:14:46.541885 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.541858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-dshm" (OuterVolumeSpecName: "dshm") pod "e8811c72-9a1f-40df-9b43-3beeead27c53" (UID: "e8811c72-9a1f-40df-9b43-3beeead27c53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:14:46.541989 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.541902    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8811c72-9a1f-40df-9b43-3beeead27c53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8811c72-9a1f-40df-9b43-3beeead27c53" (UID: "e8811c72-9a1f-40df-9b43-3beeead27c53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:14:46.541989 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.541956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8811c72-9a1f-40df-9b43-3beeead27c53-kube-api-access-47m8n" (OuterVolumeSpecName: "kube-api-access-47m8n") pod "e8811c72-9a1f-40df-9b43-3beeead27c53" (UID: "e8811c72-9a1f-40df-9b43-3beeead27c53"). InnerVolumeSpecName "kube-api-access-47m8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:14:46.641277 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.641238    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47m8n\" (UniqueName: \"kubernetes.io/projected/e8811c72-9a1f-40df-9b43-3beeead27c53-kube-api-access-47m8n\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:14:46.641277 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.641268    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8811c72-9a1f-40df-9b43-3beeead27c53-dshm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:14:46.641277 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:46.641280    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8811c72-9a1f-40df-9b43-3beeead27c53-tls-certs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Sep 09 22:14:47.167550 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.167516    2563 generic.go:356] "Generic (PLEG): container finished" podID="e8811c72-9a1f-40df-9b43-3beeead27c53" containerID="6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4" exitCode=0
Sep 09 22:14:47.167740 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.167561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"
Sep 09 22:14:47.167740 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.167608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54" event={"ID":"e8811c72-9a1f-40df-9b43-3beeead27c53","Type":"ContainerDied","Data":"6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4"}
Sep 09 22:14:47.167740 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.167653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54" event={"ID":"e8811c72-9a1f-40df-9b43-3beeead27c53","Type":"ContainerDied","Data":"6161d900904a783e1defa3a7b5f9d0eed978bff0f0bd5fa5c1f1a21b79be7135"}
Sep 09 22:14:47.167740 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.167669    2563 scope.go:117] "RemoveContainer" containerID="6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4"
Sep 09 22:14:47.176289 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.176272    2563 scope.go:117] "RemoveContainer" containerID="6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4"
Sep 09 22:14:47.176541 ip-10-0-133-204 kubenswrapper[2563]: E0909 22:14:47.176522    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4\": container with ID starting with 6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4 not found: ID does not exist" containerID="6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4"
Sep 09 22:14:47.176591 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.176552    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4"} err="failed to get container status \"6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4\": rpc error: code = NotFound desc = could not find container \"6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4\": container with ID starting with 6c2df90bc7d2e0176b601c4e37a93f64088e64a3f35e26ceb575f38e4e5930d4 not found: ID does not exist"
Sep 09 22:14:47.198383 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.198355    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"]
Sep 09 22:14:47.206146 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:47.206124    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fcc30655-kserve-7cd7fb5947-q8v54"]
Sep 09 22:14:49.046674 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:14:49.046630    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8811c72-9a1f-40df-9b43-3beeead27c53" path="/var/lib/kubelet/pods/e8811c72-9a1f-40df-9b43-3beeead27c53/volumes"
Sep 09 22:15:00.137021 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.136974    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc"]
Sep 09 22:15:00.137572 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.137551    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8811c72-9a1f-40df-9b43-3beeead27c53" containerName="main"
Sep 09 22:15:00.137667 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.137577    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8811c72-9a1f-40df-9b43-3beeead27c53" containerName="main"
Sep 09 22:15:00.137758 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.137670    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8811c72-9a1f-40df-9b43-3beeead27c53" containerName="main"
Sep 09 22:15:00.140752 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.140706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc"
Sep 09 22:15:00.144830 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.144793    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65jqq\""
Sep 09 22:15:00.149060 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.149040    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc"]
Sep 09 22:15:00.152044 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.152026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc"
Sep 09 22:15:00.423300 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:00.423272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc"]
Sep 09 22:15:00.425445 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:15:00.425403    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c9ce37_c407_42b8_8363_ddf795545d3c.slice/crio-759da3856308647c205fb60b08d029626223cb381e528bcb2f194b6a1bf25c78 WatchSource:0}: Error finding container 759da3856308647c205fb60b08d029626223cb381e528bcb2f194b6a1bf25c78: Status 404 returned error can't find the container with id 759da3856308647c205fb60b08d029626223cb381e528bcb2f194b6a1bf25c78
Sep 09 22:15:01.249039 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:01.249003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc" event={"ID":"a7c9ce37-c407-42b8-8363-ddf795545d3c","Type":"ContainerStarted","Data":"8663841f0023ccbeba7ab126c899f3fb7cfef38c54ead90277b5c869ccc33f05"}
Sep 09 22:15:01.249436 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:01.249061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc" event={"ID":"a7c9ce37-c407-42b8-8363-ddf795545d3c","Type":"ContainerStarted","Data":"759da3856308647c205fb60b08d029626223cb381e528bcb2f194b6a1bf25c78"}
Sep 09 22:15:01.264264 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:01.264212    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc" podStartSLOduration=1.264196388 podStartE2EDuration="1.264196388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:15:01.263830056 +0000 UTC m=+2506.749713095" watchObservedRunningTime="2026-09-09 22:15:01.264196388 +0000 UTC m=+2506.750079426"
Sep 09 22:15:02.256151 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:02.256114    2563 generic.go:356] "Generic (PLEG): container finished" podID="a7c9ce37-c407-42b8-8363-ddf795545d3c" containerID="8663841f0023ccbeba7ab126c899f3fb7cfef38c54ead90277b5c869ccc33f05" exitCode=0
Sep 09 22:15:02.256615 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:02.256161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc" event={"ID":"a7c9ce37-c407-42b8-8363-ddf795545d3c","Type":"ContainerDied","Data":"8663841f0023ccbeba7ab126c899f3fb7cfef38c54ead90277b5c869ccc33f05"}
Sep 09 22:15:03.396154 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:03.396126    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc"
Sep 09 22:15:04.266546 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:04.266512    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc"
Sep 09 22:15:04.266546 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:04.266535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816535-h2mzc" event={"ID":"a7c9ce37-c407-42b8-8363-ddf795545d3c","Type":"ContainerDied","Data":"759da3856308647c205fb60b08d029626223cb381e528bcb2f194b6a1bf25c78"}
Sep 09 22:15:04.266803 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:04.266589    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="759da3856308647c205fb60b08d029626223cb381e528bcb2f194b6a1bf25c78"
Sep 09 22:15:15.680991 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:15.680897    2563 scope.go:117] "RemoveContainer" containerID="6866ce6897b7f2faa6764e9f4098a670ac01bb23ca179172eb1d97daf4e9729a"
Sep 09 22:15:15.689058 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:15.689035    2563 scope.go:117] "RemoveContainer" containerID="4ccde2bfa9a02477e7abc73bed6eff8eb6be774e3c5cfad28d51853069c98fc1"
Sep 09 22:15:16.429941 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:16.429903    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658ddbb688-wdqh9_61297c74-5dc5-4627-ae8d-263c39bdd2d0/authorino/0.log"
Sep 09 22:15:22.709517 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:22.709472    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fbbd4d59-f6tcl_6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb/copy-manifests/0.log"
Sep 09 22:15:22.717019 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:22.716998    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fbbd4d59-f6tcl_6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb/manager/0.log"
Sep 09 22:15:24.045314 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:24.045285    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816535-h2mzc_a7c9ce37-c407-42b8-8363-ddf795545d3c/cleanup/0.log"
Sep 09 22:15:24.190603 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:24.190575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-d6dp4_a2bba4bd-26c6-4858-8e05-c65abf81c4d3/postgres/0.log"
Sep 09 22:15:25.065808 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:25.065773    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658ddbb688-wdqh9_61297c74-5dc5-4627-ae8d-263c39bdd2d0/authorino/0.log"
Sep 09 22:15:26.531565 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:26.531525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5769888876-6z8xd_23c1023d-671f-4ae3-85c4-e52a4860a832/kube-auth-proxy/0.log"
Sep 09 22:15:27.060374 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:27.060346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lswx7_2a06f547-5c0e-452b-8a1d-a9ab4806659c/payload-processing/0.log"
Sep 09 22:15:27.868479 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:27.868448    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-fb95q_bcb4187e-31b9-4f52-8292-9e29854fe9d9/main/0.log"
Sep 09 22:15:27.989659 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:27.989628    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-7rjdf_41ea82f9-88ef-4d5d-8c6b-9fa36e575528/main/0.log"
Sep 09 22:15:28.111057 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:28.111023    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzrf2l_2e95c85d-142e-4bbb-8618-763259a9a0bf/main/0.log"
Sep 09 22:15:28.358012 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:28.357991    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pzs8k_1ee25eb0-85cb-4f46-b0cf-b76b88743263/main/0.log"
Sep 09 22:15:37.985209 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:37.985172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7dlnc_f85f5db0-2786-426b-a8dc-b6f7d6f37fed/global-pull-secret-syncer/0.log"
Sep 09 22:15:38.151530 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:38.151498    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mnggj_cfb42f63-665c-4bd7-bf86-233e503c3533/konnectivity-agent/0.log"
Sep 09 22:15:38.199250 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:38.199210    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-204.ec2.internal_bd5a5b93ecd8465e35800ce095df071a/haproxy/0.log"
Sep 09 22:15:42.509130 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:42.509093    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658ddbb688-wdqh9_61297c74-5dc5-4627-ae8d-263c39bdd2d0/authorino/0.log"
Sep 09 22:15:44.268095 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.268065    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0fe9a7f-5caf-4e35-848e-b9d07ed13697/alertmanager/0.log"
Sep 09 22:15:44.298143 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.298114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0fe9a7f-5caf-4e35-848e-b9d07ed13697/config-reloader/0.log"
Sep 09 22:15:44.323566 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.323530    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0fe9a7f-5caf-4e35-848e-b9d07ed13697/kube-rbac-proxy-web/0.log"
Sep 09 22:15:44.347587 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.347549    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0fe9a7f-5caf-4e35-848e-b9d07ed13697/kube-rbac-proxy/0.log"
Sep 09 22:15:44.370035 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.370005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0fe9a7f-5caf-4e35-848e-b9d07ed13697/kube-rbac-proxy-metric/0.log"
Sep 09 22:15:44.391425 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.391394    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0fe9a7f-5caf-4e35-848e-b9d07ed13697/prom-label-proxy/0.log"
Sep 09 22:15:44.414011 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.413973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0fe9a7f-5caf-4e35-848e-b9d07ed13697/init-config-reloader/0.log"
Sep 09 22:15:44.499829 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.499803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bs9rc_73bca816-b6c0-43e0-bd86-4226b7b91e7c/kube-state-metrics/0.log"
Sep 09 22:15:44.524155 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.524082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bs9rc_73bca816-b6c0-43e0-bd86-4226b7b91e7c/kube-rbac-proxy-main/0.log"
Sep 09 22:15:44.545332 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.545306    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bs9rc_73bca816-b6c0-43e0-bd86-4226b7b91e7c/kube-rbac-proxy-self/0.log"
Sep 09 22:15:44.629013 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.628981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcmmm_9b72e7a2-18db-4d9d-b0c4-6eb850c905a6/node-exporter/0.log"
Sep 09 22:15:44.648014 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.647983    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcmmm_9b72e7a2-18db-4d9d-b0c4-6eb850c905a6/kube-rbac-proxy/0.log"
Sep 09 22:15:44.668267 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:44.668243    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcmmm_9b72e7a2-18db-4d9d-b0c4-6eb850c905a6/init-textfile/0.log"
Sep 09 22:15:45.235237 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.235198    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b69f4dc7-pj7rk_18c315f3-b0e1-42a9-878f-db49d6a0ffd9/telemeter-client/0.log"
Sep 09 22:15:45.266426 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.266397    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b69f4dc7-pj7rk_18c315f3-b0e1-42a9-878f-db49d6a0ffd9/reload/0.log"
Sep 09 22:15:45.292052 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.292020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b69f4dc7-pj7rk_18c315f3-b0e1-42a9-878f-db49d6a0ffd9/kube-rbac-proxy/0.log"
Sep 09 22:15:45.336537 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.336503    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d7cb6886-pp7s4_c20acab5-48d1-46c0-9d06-6f03623f6987/thanos-query/0.log"
Sep 09 22:15:45.360675 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.360605    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d7cb6886-pp7s4_c20acab5-48d1-46c0-9d06-6f03623f6987/kube-rbac-proxy-web/0.log"
Sep 09 22:15:45.382042 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.382010    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d7cb6886-pp7s4_c20acab5-48d1-46c0-9d06-6f03623f6987/kube-rbac-proxy/0.log"
Sep 09 22:15:45.406915 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.406885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d7cb6886-pp7s4_c20acab5-48d1-46c0-9d06-6f03623f6987/prom-label-proxy/0.log"
Sep 09 22:15:45.430964 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.430935    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d7cb6886-pp7s4_c20acab5-48d1-46c0-9d06-6f03623f6987/kube-rbac-proxy-rules/0.log"
Sep 09 22:15:45.453486 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:45.453459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d7cb6886-pp7s4_c20acab5-48d1-46c0-9d06-6f03623f6987/kube-rbac-proxy-metrics/0.log"
Sep 09 22:15:48.164919 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.164890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d6bff9dfc-jm2rw_17af9ed6-06c2-4774-8e81-233c07648bb2/console/0.log"
Sep 09 22:15:48.197813 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.197785    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-6q66v_44c1323a-df33-4e95-a420-f5890fa482a8/download-server/0.log"
Sep 09 22:15:48.332188 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.332150    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"]
Sep 09 22:15:48.332611 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.332591    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c9ce37-c407-42b8-8363-ddf795545d3c" containerName="cleanup"
Sep 09 22:15:48.332611 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.332611    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c9ce37-c407-42b8-8363-ddf795545d3c" containerName="cleanup"
Sep 09 22:15:48.332796 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.332698    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c9ce37-c407-42b8-8363-ddf795545d3c" containerName="cleanup"
Sep 09 22:15:48.335684 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.335663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.338333 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.338312    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j2gqd\"/\"kube-root-ca.crt\""
Sep 09 22:15:48.338439 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.338409    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j2gqd\"/\"openshift-service-ca.crt\""
Sep 09 22:15:48.339510 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.339497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j2gqd\"/\"default-dockercfg-7b6pt\""
Sep 09 22:15:48.346632 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.346611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"]
Sep 09 22:15:48.426918 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.426829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwnd\" (UniqueName: \"kubernetes.io/projected/694ebded-51ea-4870-bd16-2e75b59530a5-kube-api-access-5wwnd\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.426918 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.426867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-podres\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.427123 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.426987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-lib-modules\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.427123 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.427020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-proc\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.427123 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.427107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-sys\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528231 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-lib-modules\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528231 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-proc\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528496 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-proc\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528496 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-sys\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528496 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwnd\" (UniqueName: \"kubernetes.io/projected/694ebded-51ea-4870-bd16-2e75b59530a5-kube-api-access-5wwnd\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528496 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-podres\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528496 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-lib-modules\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528496 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-sys\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.528764 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.528515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/694ebded-51ea-4870-bd16-2e75b59530a5-podres\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.538187 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.538164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwnd\" (UniqueName: \"kubernetes.io/projected/694ebded-51ea-4870-bd16-2e75b59530a5-kube-api-access-5wwnd\") pod \"perf-node-gather-daemonset-z427n\" (UID: \"694ebded-51ea-4870-bd16-2e75b59530a5\") " pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.646404 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.646366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:48.921244 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:48.921050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"]
Sep 09 22:15:48.924305 ip-10-0-133-204 kubenswrapper[2563]: W0909 22:15:48.924280    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod694ebded_51ea_4870_bd16_2e75b59530a5.slice/crio-51cc58ecd97bad05675068352872e88bffcc0567c04aa557a8759fd71033dfbe WatchSource:0}: Error finding container 51cc58ecd97bad05675068352872e88bffcc0567c04aa557a8759fd71033dfbe: Status 404 returned error can't find the container with id 51cc58ecd97bad05675068352872e88bffcc0567c04aa557a8759fd71033dfbe
Sep 09 22:15:49.488588 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:49.488548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n" event={"ID":"694ebded-51ea-4870-bd16-2e75b59530a5","Type":"ContainerStarted","Data":"0ac941c0d17642b2a1470ce4033ad6052b90560d06f98f4ce5904a5359376564"}
Sep 09 22:15:49.489129 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:49.488637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n" event={"ID":"694ebded-51ea-4870-bd16-2e75b59530a5","Type":"ContainerStarted","Data":"51cc58ecd97bad05675068352872e88bffcc0567c04aa557a8759fd71033dfbe"}
Sep 09 22:15:49.489129 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:49.488698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:49.506813 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:49.506763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n" podStartSLOduration=1.506747729 podStartE2EDuration="1.506747729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:15:49.504271172 +0000 UTC m=+2554.990154211" watchObservedRunningTime="2026-09-09 22:15:49.506747729 +0000 UTC m=+2554.992630766"
Sep 09 22:15:50.107834 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:50.107805    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cplkq_c70d3852-c3a8-4a65-b4bc-661f6a30a9d4/dns/0.log"
Sep 09 22:15:50.128449 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:50.128423    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cplkq_c70d3852-c3a8-4a65-b4bc-661f6a30a9d4/kube-rbac-proxy/0.log"
Sep 09 22:15:50.258954 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:50.258928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lrtz6_fddb6c2d-e8e5-40a4-b0dd-74ded66c25f6/dns-node-resolver/0.log"
Sep 09 22:15:50.798419 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:50.798391    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d4fccf9d6-s9bnt_b832e8f5-9fbb-4acd-869a-9a47f01d15ad/registry/0.log"
Sep 09 22:15:50.840480 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:50.840450    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-klgv6_f3c59f67-3ceb-4cf5-a041-22852d768ac3/node-ca/0.log"
Sep 09 22:15:52.372317 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:52.372283    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5769888876-6z8xd_23c1023d-671f-4ae3-85c4-e52a4860a832/kube-auth-proxy/0.log"
Sep 09 22:15:52.499519 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:52.499484    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lswx7_2a06f547-5c0e-452b-8a1d-a9ab4806659c/payload-processing/0.log"
Sep 09 22:15:53.416917 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:53.416890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cx5mp_44420c86-87ae-49ea-9aa3-7c62ed0caf2f/serve-healthcheck-canary/0.log"
Sep 09 22:15:53.912020 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:53.911987    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxrq_9848c116-2d5e-4afd-abad-74193bc624cf/kube-rbac-proxy/0.log"
Sep 09 22:15:53.931092 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:53.931061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxrq_9848c116-2d5e-4afd-abad-74193bc624cf/exporter/0.log"
Sep 09 22:15:53.952998 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:53.952973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxrq_9848c116-2d5e-4afd-abad-74193bc624cf/extractor/0.log"
Sep 09 22:15:55.503853 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:55.503824    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j2gqd/perf-node-gather-daemonset-z427n"
Sep 09 22:15:56.822475 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:56.822442    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fbbd4d59-f6tcl_6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb/manager/0.log"
Sep 09 22:15:56.844079 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:56.844051    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fbbd4d59-f6tcl_6cf2871c-c446-4948-b9d1-8ac0b9d9d3bb/copy-manifests/0.log"
Sep 09 22:15:58.317109 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:58.316992    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-64d94cf4d-2t4b8_247d11fe-12b8-414b-b3a4-59807401b9e0/manager/0.log"
Sep 09 22:15:58.368036 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:58.368004    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4ff5n_3a3599c3-0b33-4364-b635-5ab62887d0ac/openshift-lws-operator/0.log"
Sep 09 22:15:58.905011 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:58.904980    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5769888876-6z8xd_23c1023d-671f-4ae3-85c4-e52a4860a832/kube-auth-proxy/0.log"
Sep 09 22:15:59.011684 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:15:59.011648    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lswx7_2a06f547-5c0e-452b-8a1d-a9ab4806659c/payload-processing/0.log"
Sep 09 22:16:01.177813 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:01.177700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5769888876-6z8xd_23c1023d-671f-4ae3-85c4-e52a4860a832/kube-auth-proxy/0.log"
Sep 09 22:16:01.279357 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:01.279324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lswx7_2a06f547-5c0e-452b-8a1d-a9ab4806659c/payload-processing/0.log"
Sep 09 22:16:04.880101 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:04.880064    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hx62l_0555c16f-7957-4472-a0bf-b9514e19ada1/kube-multus-additional-cni-plugins/0.log"
Sep 09 22:16:04.900034 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:04.900009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hx62l_0555c16f-7957-4472-a0bf-b9514e19ada1/egress-router-binary-copy/0.log"
Sep 09 22:16:04.923664 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:04.923617    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hx62l_0555c16f-7957-4472-a0bf-b9514e19ada1/cni-plugins/0.log"
Sep 09 22:16:04.946223 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:04.946193    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hx62l_0555c16f-7957-4472-a0bf-b9514e19ada1/bond-cni-plugin/0.log"
Sep 09 22:16:04.966467 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:04.966427    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hx62l_0555c16f-7957-4472-a0bf-b9514e19ada1/routeoverride-cni/0.log"
Sep 09 22:16:04.985218 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:04.985190    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hx62l_0555c16f-7957-4472-a0bf-b9514e19ada1/whereabouts-cni-bincopy/0.log"
Sep 09 22:16:05.003667 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:05.003636    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hx62l_0555c16f-7957-4472-a0bf-b9514e19ada1/whereabouts-cni/0.log"
Sep 09 22:16:05.400494 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:05.400462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c56fx_cc9fc8d2-e03f-44e4-89c2-2e6e697f8970/kube-multus/0.log"
Sep 09 22:16:05.423970 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:05.423935    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h2p64_0ae5a6d6-2f5b-4af6-ab7a-b629addbf014/network-metrics-daemon/0.log"
Sep 09 22:16:05.444307 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:05.444279    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h2p64_0ae5a6d6-2f5b-4af6-ab7a-b629addbf014/kube-rbac-proxy/0.log"
Sep 09 22:16:06.351574 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:06.351530    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gs2lz_8ed67432-0abd-4885-b0a8-7107e0fe1eb1/ovn-controller/0.log"
Sep 09 22:16:06.367890 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:06.367868    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gs2lz_8ed67432-0abd-4885-b0a8-7107e0fe1eb1/ovn-acl-logging/0.log"
Sep 09 22:16:06.389990 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:06.389957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gs2lz_8ed67432-0abd-4885-b0a8-7107e0fe1eb1/kube-rbac-proxy-node/0.log"
Sep 09 22:16:06.410605 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:06.410573    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gs2lz_8ed67432-0abd-4885-b0a8-7107e0fe1eb1/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 22:16:06.429663 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:06.429633    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gs2lz_8ed67432-0abd-4885-b0a8-7107e0fe1eb1/northd/0.log"
Sep 09 22:16:06.450187 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:06.450163    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gs2lz_8ed67432-0abd-4885-b0a8-7107e0fe1eb1/nbdb/0.log"
Sep 09 22:16:06.472175 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:06.472147    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gs2lz_8ed67432-0abd-4885-b0a8-7107e0fe1eb1/sbdb/0.log"
Sep 09 22:16:06.532761 ip-10-0-133-204 kubenswrapper[2563]: I0909 22:16:06.532703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gs2lz_8ed67432-0abd-4885-b0a8-7107e0fe1eb1/ovnkube-controller/0.log"