Sep 09 03:05:50.373154 ip-10-0-143-233 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 03:05:50.771735 ip-10-0-143-233 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:50.771735 ip-10-0-143-233 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 03:05:50.771735 ip-10-0-143-233 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:50.771735 ip-10-0-143-233 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 03:05:50.771735 ip-10-0-143-233 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:05:50.774358 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.774265    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 03:05:50.779177 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779159    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.779177 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779176    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779182    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779186    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779189    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779192    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779195    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779198    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779201    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779203    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779205    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779209    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779217    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779222    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779225    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779228    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779231    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779233    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779236    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779239    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779241    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.779248 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779244    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779247    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779250    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779253    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779255    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779258    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779261    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779264    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779267    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779269    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779271    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779274    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779277    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779279    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779282    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779286    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779289    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779292    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779295    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779297    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.779736 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779299    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779302    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779304    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779307    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779309    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779312    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779314    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779332    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779335    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779337    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779340    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779342    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779345    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779349    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779351    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779354    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779357    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779360    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779364    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779367    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.780231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779370    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779374    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779376    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779379    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779382    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779386    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779388    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779391    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779393    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779396    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779399    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779402    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779405    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779407    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779410    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779412    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779414    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779417    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779420    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.780759 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779422    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779425    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779427    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779440    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779443    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779446    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779450    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779452    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779455    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779459    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779463    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779466    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779468    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779471    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779474    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779477    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779480    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779483    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.779486    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.781231 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780204    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780210    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780214    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780217    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780220    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780223    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780226    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780229    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780232    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780234    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780237    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780240    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780243    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780245    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780248    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780251    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780253    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780256    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780259    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780262    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.781687 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780265    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780268    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780271    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780274    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780276    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780279    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780282    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780284    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780287    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780289    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780293    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780295    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780298    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780300    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780302    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780305    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780307    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780310    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780313    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780315    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.782259 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780317    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780320    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780323    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780326    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780328    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780331    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780334    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780336    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780339    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780341    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780346    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780349    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780352    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780354    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780357    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780359    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780362    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780365    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780367    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780370    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.782798 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780372    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780375    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780377    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780380    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780383    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780385    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780387    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780390    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780392    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780395    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780397    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780400    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780403    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780406    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780408    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780410    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780414    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780418    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780421    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.783319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780425    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780427    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780430    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780434    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780437    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780439    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780442    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780444    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780448    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780451    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780454    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780457    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780460    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780464    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780467    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780470    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780473    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780476    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780478    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.783782 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780481    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780873    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780878    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780881    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780884    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780889    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780893    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780896    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780899    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780902    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780905    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780907    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780910    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780912    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780915    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780918    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780923    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780926    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780929    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.784268 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780931    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780934    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780936    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780939    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780941    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780944    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780946    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780950    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780954    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780957    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780959    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780962    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780965    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780967    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780970    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780972    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780974    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780977    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780979    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780982    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.784720 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780984    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780987    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780989    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780992    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780994    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780997    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.780999    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781001    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781007    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781011    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781014    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781016    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781019    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781021    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781023    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781026    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781029    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781031    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781034    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.785214 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781036    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781039    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781041    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781044    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781046    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781048    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781051    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781055    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781058    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781060    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781062    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781065    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781067    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781070    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781073    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781075    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781078    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781080    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781083    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781085    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.785683 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781088    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781091    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781095    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781098    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781100    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781103    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781106    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781108    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781111    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781113    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781115    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781118    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781120    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781123    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781125    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781128    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781130    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781132    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781135    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781138    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.786202 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781140    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781142    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781218    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781253    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781265    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781269    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781274    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781277    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781282    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781287    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781290    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781293    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781296    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781301    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781306    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781309    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781312    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781315    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781319    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781322    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781325    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781328    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781331    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781334    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 03:05:50.786670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781337    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781341    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781344    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781347    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781351    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781355    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781357    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781360    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781363    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781366    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781371    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781374    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781377    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781380    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781383    2583 flags.go:64] FLAG: --enable-server="true"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781387    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781391    2583 flags.go:64] FLAG: --event-burst="100"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781394    2583 flags.go:64] FLAG: --event-qps="50"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781397    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781400    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781403    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781408    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781412    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781415    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781419    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 09 03:05:50.787249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781422    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781425    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781428    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781430    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781433    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781436    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781439    2583 flags.go:64] FLAG: --feature-gates=""
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781443    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781446    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781449    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781452    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781455    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781459    2583 flags.go:64] FLAG: --help="false"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781462    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-143-233.ec2.internal"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781465    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781467    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781470    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781473    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781477    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781479    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781483    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781485    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781488    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 03:05:50.787870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781492    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781495    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781497    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781500    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781503    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781507    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781512    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781515    2583 flags.go:64] FLAG: --lock-file=""
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781518    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781522    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781525    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781530    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781533    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781536    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781538    2583 flags.go:64] FLAG: --logging-format="text"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781541    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781545    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781547    2583 flags.go:64] FLAG: --manifest-url=""
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781550    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781555    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781558    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781562    2583 flags.go:64] FLAG: --max-pods="110"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781565    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781568    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781571    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 03:05:50.788482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781574    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781577    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781580    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781582    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781590    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781593    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781597    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781600    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781603    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781609    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781612    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781615    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781618    2583 flags.go:64] FLAG: --port="10250"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781623    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781626    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a9703e90dfdd9f72"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781630    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781632    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781635    2583 flags.go:64] FLAG: --register-node="true"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781638    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781642    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781645    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781648    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781650    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781654    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781657    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 03:05:50.789130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781660    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781663    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781665    2583 flags.go:64] FLAG: --runonce="false"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781668    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781671    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781674    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781677    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781680    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781683    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781686    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781689    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781692    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781695    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781698    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781701    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781704    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781707    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781710    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781715    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781718    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781722    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781726    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781728    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781731    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781734    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 03:05:50.789908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781737    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781740    2583 flags.go:64] FLAG: --v="2"
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781744    2583 flags.go:64] FLAG: --version="false"
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781748    2583 flags.go:64] FLAG: --vmodule=""
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781751    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.781754    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781909    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781913    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781916    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781919    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781922    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781926    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781928    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781931    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781934    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781937    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781940    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781942    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781944    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781947    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781949    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.790908 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781952    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781954    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781957    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781960    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781963    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781965    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781968    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781971    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781974    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781977    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781979    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781982    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781984    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781987    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781989    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781992    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781994    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781996    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.781999    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782001    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.791518 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782004    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782006    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782009    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782012    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782014    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782016    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782019    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782021    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782024    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782026    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782029    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782032    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782034    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782036    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782039    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782041    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782044    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782047    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782049    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782052    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.792042 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782055    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782057    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782060    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782064    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782067    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782070    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782073    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782075    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782078    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782081    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782084    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782088    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782091    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782095    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782097    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782100    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782103    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782106    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782109    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782111    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.792545 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782114    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782116    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782119    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782122    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782124    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782127    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782129    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782131    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782134    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782137    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782140    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782142    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782144    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782147    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782150    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782152    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782154    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782157    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782159    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.793071 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782163    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782166    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782168    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782171    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.782173    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.782178    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.790317    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.790353    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.791100    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.791193    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791445    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791455    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791461    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791466    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.793523 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791471    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791475    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791479    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791483    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791489    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791493    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791497    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791501    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791505    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791510    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791515    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791519    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791524    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791528    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791533    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791537    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791542    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791547    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791556    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791564    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.793916 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791571    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791577    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791584    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791588    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791594    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791598    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791605    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791615    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791618    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791621    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791623    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791626    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791630    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791633    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791636    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791638    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791641    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791643    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791646    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.794400 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791650    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791654    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791657    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791660    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791663    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791666    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791668    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791671    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791674    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791677    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791680    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791682    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791684    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791687    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791690    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791692    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791694    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791697    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791699    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791702    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.794880 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791704    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791707    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791710    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791712    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791714    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791717    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791720    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791723    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791725    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791728    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791730    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791732    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791735    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791737    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791740    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791742    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791745    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791748    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791751    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791753    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.795361 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791755    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791758    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791761    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791763    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791766    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791768    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791771    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791773    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791775    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791778    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791780    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791782    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791785    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791787    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791790    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.795900 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791792    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.791798    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791939    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791944    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791947    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791950    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791953    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791955    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791958    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791961    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791963    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791966    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791969    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791971    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791982    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:05:50.796265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791985    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791987    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791990    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791992    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791996    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.791998    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792000    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792003    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792005    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792008    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792010    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792013    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792016    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792018    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792021    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792025    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792027    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792029    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792032    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:05:50.796625 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792034    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792037    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792039    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792041    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792044    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792046    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792049    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792052    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792054    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792057    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792060    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792063    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792065    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792073    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792076    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792078    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792081    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792083    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792086    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792088    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:05:50.797144 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792091    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792093    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792096    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792098    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792101    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792103    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792105    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792108    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792110    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792113    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792115    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792117    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792120    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792122    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792125    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792127    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792130    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792134    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792137    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:05:50.797626 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792140    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792143    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792146    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792149    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792152    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792155    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792157    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792165    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792168    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792170    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792173    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792175    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792178    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792180    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792183    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792185    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792187    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792190    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792193    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:05:50.798140 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792195    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792198    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792200    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792203    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792205    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792207    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792210    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792212    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:50.792216    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.792221    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.792367    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.796748    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.797728    2583 server.go:1019] "Starting client certificate rotation"
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.797838    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:05:50.798632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.797874    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:05:50.801067 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.801053    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:05:50.802248 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.802231    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:05:50.812281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.812255    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 09 03:05:50.815248 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.815231    2583 log.go:25] "Validated CRI v1 image API"
Sep 09 03:05:50.816301 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.816274    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 03:05:50.820817 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.820797    2583 fs.go:135] Filesystem UUIDs: map[56662a7c-f3f9-473b-94e3-373992614ebf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b0a1372f-1817-46c2-8f83-fadba8c3f228:/dev/nvme0n1p3]
Sep 09 03:05:50.820924 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.820815    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 03:05:50.826518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.826405    2583 manager.go:216] Machine: {Timestamp:2026-09-09 03:05:50.825196542 +0000 UTC m=+0.339278703 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3133828 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b21ef9edc8c41be1d19558fb0d2b8 SystemUUID:ec2b21ef-9edc-8c41-be1d-19558fb0d2b8 BootID:adc172f3-c7ff-4abe-932b-80d939288446 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:27:d4:7f:00:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:27:d4:7f:00:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:8a:25:1f:3c:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 03:05:50.826518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.826514    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 03:05:50.826620 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.826603    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 03:05:50.828096 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.828081    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:05:50.829668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.829638    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 03:05:50.829844 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.829674    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 03:05:50.830588 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.830578    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 03:05:50.830625 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.830592    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 03:05:50.830625 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.830600    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:05:50.831543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.831531    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:05:50.832766 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.832755    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 03:05:50.835551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.835539    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:05:50.835685 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.835675    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 03:05:50.835736 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.835728    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 03:05:50.835768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.835748    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 03:05:50.835768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.835762    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 03:05:50.835824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.835775    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 09 03:05:50.835824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.835784    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 03:05:50.837624 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.837557    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:05:50.837624 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.837594    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:05:50.840603 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.840586    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 03:05:50.842874 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.842860    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 03:05:50.842927 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.842890    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 03:05:50.844258 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844246    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 03:05:50.844304 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844265    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 03:05:50.844304 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844271    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 03:05:50.844304 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844277    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 03:05:50.844304 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844286    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 03:05:50.844304 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844291    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 03:05:50.844304 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844305    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 03:05:50.844455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844310    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 03:05:50.844455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844317    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 03:05:50.844455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844322    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 03:05:50.844455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844330    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 03:05:50.844455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.844334    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 03:05:50.845683 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.845668    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 03:05:50.845725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.845687    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 03:05:50.847643 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.847621    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 03:05:50.847721 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.847626    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 03:05:50.849616 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.849603    2583 server.go:1268] "Started kubelet"
Sep 09 03:05:50.849719 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.849691    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 03:05:50.849775 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.849715    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 03:05:50.849819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.849797    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 03:05:50.850551 ip-10-0-143-233 systemd[1]: Started Kubernetes Kubelet.
Sep 09 03:05:50.851919 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.850866    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 03:05:50.853928 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.853913    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 03:05:50.855295 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.855269    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 03:05:50.857411 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.857377    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 03:05:50.858116 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.858087    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-233.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 03:05:50.858774 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.858757    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 03:05:50.859165 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.858066    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-233.ec2.internal.18d38820a8dc819d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-233.ec2.internal,UID:ip-10-0-143-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-233.ec2.internal,},FirstTimestamp:2026-09-09 03:05:50.849573277 +0000 UTC m=+0.363655419,LastTimestamp:2026-09-09 03:05:50.849573277 +0000 UTC m=+0.363655419,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-233.ec2.internal,}"
Sep 09 03:05:50.859443 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859418    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 03:05:50.859443 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859442    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 03:05:50.859551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859512    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 03:05:50.859551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859523    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 03:05:50.859638 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859570    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 03:05:50.859691 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859673    2583 factory.go:55] Registering systemd factory
Sep 09 03:05:50.859741 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859709    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 09 03:05:50.859988 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859970    2583 factory.go:153] Registering CRI-O factory
Sep 09 03:05:50.859988 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.859990    2583 factory.go:223] Registration of the crio container factory successfully
Sep 09 03:05:50.860102 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.860042    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 03:05:50.860102 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.860073    2583 factory.go:103] Registering Raw factory
Sep 09 03:05:50.860102 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.860090    2583 manager.go:1196] Started watching for new ooms in manager
Sep 09 03:05:50.861531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.861506    2583 manager.go:349] Starting recovery of all containers
Sep 09 03:05:50.861736 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.861648    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:50.864588 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.864566    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 03:05:50.864672 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.864595    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 03:05:50.872732 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.872716    2583 manager.go:354] Recovery completed
Sep 09 03:05:50.875231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.875211    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pbzfs"
Sep 09 03:05:50.876810 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.876794    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.878734 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.878719    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.878808 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.878747    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.878808 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.878757    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.879297 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.879283    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 03:05:50.879297 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.879295    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 03:05:50.879413 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.879307    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:50.879413 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.879309    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:05:50.880268 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.880253    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:50.880325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.880277    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:50.880325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.880292    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:50.880560 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.880546    2583 policy_none.go:49] "None policy: Start"
Sep 09 03:05:50.880560 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.880563    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 03:05:50.880727 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.880572    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 03:05:50.880727 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.880525    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-233.ec2.internal.18d38820aa997971  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-233.ec2.internal,UID:ip-10-0-143-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-233.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-233.ec2.internal,},FirstTimestamp:2026-09-09 03:05:50.878734705 +0000 UTC m=+0.392816846,LastTimestamp:2026-09-09 03:05:50.878734705 +0000 UTC m=+0.392816846,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-233.ec2.internal,}"
Sep 09 03:05:50.881600 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.881588    2583 policy_none.go:47] "Start"
Sep 09 03:05:50.882509 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.882493    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pbzfs"
Sep 09 03:05:50.915594 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.915578    2583 manager.go:337] "Starting Device Plugin manager"
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.915630    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.915646    2583 server.go:87] "Starting device plugin registration server"
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.916064    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.916078    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.916174    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.916245    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.916254    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.916856    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 03:05:50.931893 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:50.916902    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:50.999338 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:50.999299    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 03:05:51.000625 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.000610    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 03:05:51.000712 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.000664    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 03:05:51.000712 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.000693    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 03:05:51.000792 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.000740    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 03:05:51.003235 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.003214    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:51.016951 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.016926    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:51.017795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.017779    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:51.017905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.017821    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:51.017905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.017858    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:51.017905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.017892    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.027346 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.027265    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.027346 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.027289    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-233.ec2.internal\": node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.050972 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.050945    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.101678 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.101627    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"]
Sep 09 03:05:51.101884 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.101729    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:51.103381 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.103357    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:51.103495 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.103391    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:51.103495 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.103403    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:51.104689 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.104676    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:51.104851 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.104821    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.104906 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.104897    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:51.105465 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.105445    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:51.105572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.105476    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:51.105572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.105486    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:51.105572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.105452    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:51.105572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.105550    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:51.105572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.105561    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:51.106567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.106550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.106653 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.106579    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:05:51.107284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.107268    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:05:51.107379 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.107298    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:05:51.107379 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.107309    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:05:51.131630 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.131603    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-233.ec2.internal\" not found" node="ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.136141 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.136124    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-233.ec2.internal\" not found" node="ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.152048 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.152029    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.252660 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.252608    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.260940 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.260917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aecef6bfb645f61d4d21f56287e6cd97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal\" (UID: \"aecef6bfb645f61d4d21f56287e6cd97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.261030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.260944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecef6bfb645f61d4d21f56287e6cd97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal\" (UID: \"aecef6bfb645f61d4d21f56287e6cd97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.261030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.260975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f4776863423ca28036bea3b8e2635ca-config\") pod \"kube-apiserver-proxy-ip-10-0-143-233.ec2.internal\" (UID: \"3f4776863423ca28036bea3b8e2635ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.353405 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.353331    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.361728 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.361698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aecef6bfb645f61d4d21f56287e6cd97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal\" (UID: \"aecef6bfb645f61d4d21f56287e6cd97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.361792 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.361731    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecef6bfb645f61d4d21f56287e6cd97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal\" (UID: \"aecef6bfb645f61d4d21f56287e6cd97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.361792 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.361748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f4776863423ca28036bea3b8e2635ca-config\") pod \"kube-apiserver-proxy-ip-10-0-143-233.ec2.internal\" (UID: \"3f4776863423ca28036bea3b8e2635ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.361792 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.361774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f4776863423ca28036bea3b8e2635ca-config\") pod \"kube-apiserver-proxy-ip-10-0-143-233.ec2.internal\" (UID: \"3f4776863423ca28036bea3b8e2635ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.361920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.361796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aecef6bfb645f61d4d21f56287e6cd97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal\" (UID: \"aecef6bfb645f61d4d21f56287e6cd97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.361920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.361802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecef6bfb645f61d4d21f56287e6cd97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal\" (UID: \"aecef6bfb645f61d4d21f56287e6cd97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.433907 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.433875    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.439082 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.439063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:51.453866 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.453843    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.555007 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.554975    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.655613 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.655528    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.756161 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.756125    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.797535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.797498    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 03:05:51.798039 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.797619    2583 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.143.233:57404->98.89.212.114:6443: use of closed network connection"
Sep 09 03:05:51.798039 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.797639    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:05:51.798039 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.797706    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: unexpected error when reading response body. Please retry. Original error: read tcp 10.0.143.233:57404->98.89.212.114:6443: use of closed network connection" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 03:05:51.857171 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.857132    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:51.884943 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.884900    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 03:00:51 +0000 UTC" deadline="2028-06-17 03:44:39.386240726 +0000 UTC"
Sep 09 03:05:51.884943 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:51.884932    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15528h38m47.501313845s"
Sep 09 03:05:51.957381 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:51.957334    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.033062 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:52.033036    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:52.058391 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.058350    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.071413 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:52.071389    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:52.107674 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:52.107637    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4776863423ca28036bea3b8e2635ca.slice/crio-6f13057a1f4445538d430b14cb29c3feef397c2b291be60de14e71a566482a08 WatchSource:0}: Error finding container 6f13057a1f4445538d430b14cb29c3feef397c2b291be60de14e71a566482a08: Status 404 returned error can't find the container with id 6f13057a1f4445538d430b14cb29c3feef397c2b291be60de14e71a566482a08
Sep 09 03:05:52.107967 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:52.107940    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecef6bfb645f61d4d21f56287e6cd97.slice/crio-18305dad33d2f03317024e687df96c023eee73f7c33991ff2bab101e0f90cec3 WatchSource:0}: Error finding container 18305dad33d2f03317024e687df96c023eee73f7c33991ff2bab101e0f90cec3: Status 404 returned error can't find the container with id 18305dad33d2f03317024e687df96c023eee73f7c33991ff2bab101e0f90cec3
Sep 09 03:05:52.112573 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:52.112558    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:05:52.160714 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.160666    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.261771 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.261701    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.361979 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.361947    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.466206 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.466179    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.566503 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.566409    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.666533 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.666497    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.767390 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.767353    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.867741 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.867654    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:52.968898 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:52.968692    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.008172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:53.008079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal" event={"ID":"aecef6bfb645f61d4d21f56287e6cd97","Type":"ContainerStarted","Data":"18305dad33d2f03317024e687df96c023eee73f7c33991ff2bab101e0f90cec3"}
Sep 09 03:05:53.011896 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:53.010398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal" event={"ID":"3f4776863423ca28036bea3b8e2635ca","Type":"ContainerStarted","Data":"6f13057a1f4445538d430b14cb29c3feef397c2b291be60de14e71a566482a08"}
Sep 09 03:05:53.069384 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.069340    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.170245 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.170158    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.260002 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:53.259964    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:53.271133 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.271098    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.371260 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.371219    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.471967 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.471882    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.572107 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.572067    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.672932 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.672891    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.773861 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.773750    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.873916 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.873880    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:53.974780 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:53.974738    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.075098 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.075017    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.175717 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.175676    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.276222 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.276195    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.377143 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.377114    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.477914 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.477880    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.578266 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.578219    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.679425 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.679340    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.779698 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.779655    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.880987 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:54.880950    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-233.ec2.internal\" not found"
Sep 09 03:05:54.922030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:54.922003    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:05:54.961723 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:54.961640    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:54.975497 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:54.975475    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:54.977668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:54.977655    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:54.986322 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:54.986304    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:55.016201 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.016171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal" event={"ID":"aecef6bfb645f61d4d21f56287e6cd97","Type":"ContainerStarted","Data":"ea2a3fa30252f2e732178f5af476737261198d2aea26561c6892e33ace63e2a1"}
Sep 09 03:05:55.016406 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.016392    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:55.017810 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.017785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal" event={"ID":"3f4776863423ca28036bea3b8e2635ca","Type":"ContainerStarted","Data":"b7c9c4dcd1c0a8ace44f3c918eede3943d55a20dfb0b5d992f7587db057c3858"}
Sep 09 03:05:55.018000 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.017988    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:55.028785 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.028764    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:55.028868 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:55.028810    2583 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-233.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:55.029127 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.029113    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:55.029170 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:55.029152    2583 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:55.841067 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.841034    2583 apiserver.go:52] "Watching apiserver"
Sep 09 03:05:55.851985 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.851963    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 03:05:55.853045 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.853023    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-2p8lk","openshift-multus/multus-additional-cni-plugins-9w82s","openshift-multus/network-metrics-daemon-zqzm9","openshift-cluster-node-tuning-operator/tuned-28jf7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal","openshift-multus/multus-bccdk","openshift-network-diagnostics/network-check-target-nkqd9","openshift-network-operator/iptables-alerter-k774j","openshift-ovn-kubernetes/ovnkube-node-94lfn","kube-system/global-pull-secret-syncer-g698z","kube-system/konnectivity-agent-x4mml","kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm","openshift-dns/node-resolver-9zhgm"]
Sep 09 03:05:55.854548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.854531    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:05:55.854621 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:55.854603    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:05:55.855671 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.855649    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:05:55.856885 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.856869    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:55.856969 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:55.856944    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:05:55.858280 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.858198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.858392 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.858292    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5qp5q\""
Sep 09 03:05:55.858392 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.858320    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 03:05:55.858637 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.858618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 03:05:55.859762 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.859747    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.860298 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.860279    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 03:05:55.861021 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.860996    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xhrgh\""
Sep 09 03:05:55.861121 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.861024    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 03:05:55.861121 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.861006    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:05:55.861121 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.861008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.862005 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.861985    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 03:05:55.862088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.862046    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l9htr\""
Sep 09 03:05:55.862088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.862048    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 03:05:55.862495 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.862467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.865059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.864152    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 03:05:55.865059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.864250    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 03:05:55.865059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.864500    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l87bb\""
Sep 09 03:05:55.865059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.864654    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 03:05:55.865059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.864855    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 03:05:55.865059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.865017    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 03:05:55.865446 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.865202    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 03:05:55.865446 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.865236    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:55.865446 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.865440    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 03:05:55.866596 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.866578    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 03:05:55.866713 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.866610    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 03:05:55.866713 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.866692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbgq6\""
Sep 09 03:05:55.867163 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.867145    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 03:05:55.867968 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.867519    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:55.867968 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.867682    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 03:05:55.867968 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.867704    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 03:05:55.868142 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.867994    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 03:05:55.868453 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.868438    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-867vf\""
Sep 09 03:05:55.869422 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.869403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:55.869506 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:55.869452    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:05:55.869615 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.869600    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hmw4t\""
Sep 09 03:05:55.870019 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.870000    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 03:05:55.870120 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.870026    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 03:05:55.871197 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.871182    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:55.872693 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.872675    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:55.873220 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.873200    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:05:55.873535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.873450    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 03:05:55.873535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.873460    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 03:05:55.873535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.873528    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5w9fz\""
Sep 09 03:05:55.873697 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.873574    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 03:05:55.874986 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.874959    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 03:05:55.875086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.875066    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lchdn\""
Sep 09 03:05:55.875190 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.875173    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:05:55.875249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.875224    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 03:05:55.889649 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.889624    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hdbnq"
Sep 09 03:05:55.892848 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.892827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2tq\" (UniqueName: \"kubernetes.io/projected/0c3c44b2-531b-4a24-8daf-a05142141da9-kube-api-access-8g2tq\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.892976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.892858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-cni-bin\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.892976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.892875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-conf-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.892976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.892891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysconfig\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.892976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.892930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-systemd\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.892976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.892969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/742fbb48-0057-4d26-9b3c-2493b94a952e-hosts-file\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-cni-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-hostroot\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysctl-conf\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-sys\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-socket-dir-parent\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fae6eb0-381c-4cc5-9098-8c95abeca3ba-konnectivity-ca\") pod \"konnectivity-agent-x4mml\" (UID: \"4fae6eb0-381c-4cc5-9098-8c95abeca3ba\") " pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893134    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-run\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-netns\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-cni-multus\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-multus-certs\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-k8s-cni-cncf-io\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbhtb\" (UniqueName: \"kubernetes.io/projected/734355ce-8feb-4305-8872-543d37a47fc4-kube-api-access-hbhtb\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-var-lib-kubelet\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqf2r\" (UniqueName: \"kubernetes.io/projected/742fbb48-0057-4d26-9b3c-2493b94a952e-kube-api-access-tqf2r\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fae6eb0-381c-4cc5-9098-8c95abeca3ba-agent-certs\") pod \"konnectivity-agent-x4mml\" (UID: \"4fae6eb0-381c-4cc5-9098-8c95abeca3ba\") " pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-modprobe-d\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893330    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-lib-modules\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-host\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c3c44b2-531b-4a24-8daf-a05142141da9-tmp\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/742fbb48-0057-4d26-9b3c-2493b94a952e-tmp-dir\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-system-cni-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-kubelet\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39f5888a-646d-4f07-9073-aebd7f316182-multus-daemon-config\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-kubernetes\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.893586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-cnibin\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.894203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39f5888a-646d-4f07-9073-aebd7f316182-cni-binary-copy\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.894203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-etc-kubernetes\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.894203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxg7\" (UniqueName: \"kubernetes.io/projected/39f5888a-646d-4f07-9073-aebd7f316182-kube-api-access-8jxg7\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.894203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:05:55.894203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysctl-d\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.894203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.893687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-tuned\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.897681 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.897662    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hdbnq"
Sep 09 03:05:55.960384 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.960352    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 03:05:55.994482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxgm\" (UniqueName: \"kubernetes.io/projected/7d565593-7fbb-41ef-8856-8b6dfa88a62c-kube-api-access-jjxgm\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:55.994667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-etc-selinux\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:55.994667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a30635a-06fa-47d4-9416-be835f7efd0a-iptables-alerter-script\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:55.994667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-cni-multus\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.994667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbhtb\" (UniqueName: \"kubernetes.io/projected/734355ce-8feb-4305-8872-543d37a47fc4-kube-api-access-hbhtb\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:55.994667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-var-lib-kubelet\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.994667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-env-overrides\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.994667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e353f605-9075-414d-aa37-b8b53f094c02-kubelet-config\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-var-lib-kubelet\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994676    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-host\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-cni-multus\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c3c44b2-531b-4a24-8daf-a05142141da9-tmp\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cnibin\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-host\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-kubelet\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h85s\" (UniqueName: \"kubernetes.io/projected/9b96da89-37da-4ea3-a3f6-33282f05e059-kube-api-access-2h85s\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-system-cni-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-kubelet\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994898    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-system-cni-dir\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994938    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-kubelet\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnzm\" (UniqueName: \"kubernetes.io/projected/6b7fca20-9b25-4a26-a49d-3089acb2966e-kube-api-access-9nnzm\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.994986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a30635a-06fa-47d4-9416-be835f7efd0a-host-slash\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39f5888a-646d-4f07-9073-aebd7f316182-cni-binary-copy\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-etc-kubernetes\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-system-cni-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxg7\" (UniqueName: \"kubernetes.io/projected/39f5888a-646d-4f07-9073-aebd7f316182-kube-api-access-8jxg7\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995116    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-etc-kubernetes\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995143    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-tuned\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2tq\" (UniqueName: \"kubernetes.io/projected/0c3c44b2-531b-4a24-8daf-a05142141da9-kube-api-access-8g2tq\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-systemd-units\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-cni-bin\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-conf-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-var-lib-cni-bin\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-slash\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995602    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-etc-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-conf-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-cni-bin\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.995898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995656    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39f5888a-646d-4f07-9073-aebd7f316182-cni-binary-copy\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovn-node-metrics-cert\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-registration-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/742fbb48-0057-4d26-9b3c-2493b94a952e-hosts-file\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995779    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-cni-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-hostroot\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-sys\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cni-binary-copy\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/742fbb48-0057-4d26-9b3c-2493b94a952e-hosts-file\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995862    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-hostroot\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-device-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-sys-fs\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995911    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-sys\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-socket-dir-parent\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-cni-dir\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fae6eb0-381c-4cc5-9098-8c95abeca3ba-konnectivity-ca\") pod \"konnectivity-agent-x4mml\" (UID: \"4fae6eb0-381c-4cc5-9098-8c95abeca3ba\") " pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995991    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-multus-socket-dir-parent\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.996866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.995998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b96da89-37da-4ea3-a3f6-33282f05e059-host\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-netns\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-multus-certs\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-netns\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovnkube-config\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-multus-certs\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b96da89-37da-4ea3-a3f6-33282f05e059-serviceca\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-k8s-cni-cncf-io\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-run-netns\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996257    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-cni-netd\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-host-run-k8s-cni-cncf-io\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-var-lib-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqf2r\" (UniqueName: \"kubernetes.io/projected/742fbb48-0057-4d26-9b3c-2493b94a952e-kube-api-access-tqf2r\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fae6eb0-381c-4cc5-9098-8c95abeca3ba-agent-certs\") pod \"konnectivity-agent-x4mml\" (UID: \"4fae6eb0-381c-4cc5-9098-8c95abeca3ba\") " pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-modprobe-d\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-lib-modules\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.998129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fae6eb0-381c-4cc5-9098-8c95abeca3ba-konnectivity-ca\") pod \"konnectivity-agent-x4mml\" (UID: \"4fae6eb0-381c-4cc5-9098-8c95abeca3ba\") " pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-modprobe-d\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovnkube-script-lib\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e353f605-9075-414d-aa37-b8b53f094c02-dbus\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/742fbb48-0057-4d26-9b3c-2493b94a952e-tmp-dir\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39f5888a-646d-4f07-9073-aebd7f316182-multus-daemon-config\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-lib-modules\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:55.996695    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-kubernetes\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:55.996792    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:56.496750785 +0000 UTC m=+6.010832928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-kubernetes\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/742fbb48-0057-4d26-9b3c-2493b94a952e-tmp-dir\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-node-log\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996927    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-run-ovn-kubernetes\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-socket-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:55.999030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.996983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-cnibin\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysctl-d\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-ovn\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-log-socket\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysctl-d\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39f5888a-646d-4f07-9073-aebd7f316182-cnibin\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysconfig\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-systemd\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysconfig\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-systemd\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r2hv\" (UniqueName: \"kubernetes.io/projected/b6f86368-5761-4e43-bb5b-022524a4d82c-kube-api-access-7r2hv\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28c8l\" (UniqueName: \"kubernetes.io/projected/2a30635a-06fa-47d4-9416-be835f7efd0a-kube-api-access-28c8l\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysctl-conf\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-systemd\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-run\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:55.999920 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-run\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:56.000425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-sysctl-conf\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:56.000425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:55.997783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39f5888a-646d-4f07-9073-aebd7f316182-multus-daemon-config\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:56.000425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.000099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fae6eb0-381c-4cc5-9098-8c95abeca3ba-agent-certs\") pod \"konnectivity-agent-x4mml\" (UID: \"4fae6eb0-381c-4cc5-9098-8c95abeca3ba\") " pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:05:56.000518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.000501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c3c44b2-531b-4a24-8daf-a05142141da9-tmp\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:56.000606 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.000580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c3c44b2-531b-4a24-8daf-a05142141da9-etc-tuned\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:56.002364 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.002347    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:56.002416 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.002368    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:56.002416 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.002380    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dnwqj for pod openshift-network-diagnostics/network-check-target-nkqd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:56.002480 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.002447    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj podName:bafde9e9-c0d3-4b04-b0eb-d77c722be9ec nodeName:}" failed. No retries permitted until 2026-09-09 03:05:56.502431888 +0000 UTC m=+6.016514020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dnwqj" (UniqueName: "kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj") pod "network-check-target-nkqd9" (UID: "bafde9e9-c0d3-4b04-b0eb-d77c722be9ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:56.005025 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.005003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbhtb\" (UniqueName: \"kubernetes.io/projected/734355ce-8feb-4305-8872-543d37a47fc4-kube-api-access-hbhtb\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:56.005136 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.005114    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2tq\" (UniqueName: \"kubernetes.io/projected/0c3c44b2-531b-4a24-8daf-a05142141da9-kube-api-access-8g2tq\") pod \"tuned-28jf7\" (UID: \"0c3c44b2-531b-4a24-8daf-a05142141da9\") " pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:56.005242 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.005216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxg7\" (UniqueName: \"kubernetes.io/projected/39f5888a-646d-4f07-9073-aebd7f316182-kube-api-access-8jxg7\") pod \"multus-bccdk\" (UID: \"39f5888a-646d-4f07-9073-aebd7f316182\") " pod="openshift-multus/multus-bccdk"
Sep 09 03:05:56.009300 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.009278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqf2r\" (UniqueName: \"kubernetes.io/projected/742fbb48-0057-4d26-9b3c-2493b94a952e-kube-api-access-tqf2r\") pod \"node-resolver-9zhgm\" (UID: \"742fbb48-0057-4d26-9b3c-2493b94a952e\") " pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:56.019843 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.019824    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:56.020006 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.019991    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:56.032450 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.032422    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:56.032551 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.032467    2583 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-233.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:56.035665 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.035649    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:05:56.035776 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.035688    2583 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal"
Sep 09 03:05:56.084907 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.084836    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-233.ec2.internal" podStartSLOduration=1.084822351 podStartE2EDuration="1.084822351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:05:56.084595063 +0000 UTC m=+5.598677218" watchObservedRunningTime="2026-09-09 03:05:56.084822351 +0000 UTC m=+5.598904504"
Sep 09 03:05:56.098969 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.098901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-env-overrides\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.098969 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.098931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e353f605-9075-414d-aa37-b8b53f094c02-kubelet-config\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:56.098969 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.098955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cnibin\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.098978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.098998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-kubelet\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h85s\" (UniqueName: \"kubernetes.io/projected/9b96da89-37da-4ea3-a3f6-33282f05e059-kube-api-access-2h85s\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-system-cni-dir\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099073    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cnibin\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnzm\" (UniqueName: \"kubernetes.io/projected/6b7fca20-9b25-4a26-a49d-3089acb2966e-kube-api-access-9nnzm\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-kubelet\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a30635a-06fa-47d4-9416-be835f7efd0a-host-slash\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a30635a-06fa-47d4-9416-be835f7efd0a-host-slash\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:56.099176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e353f605-9075-414d-aa37-b8b53f094c02-kubelet-config\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-system-cni-dir\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-systemd-units\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-slash\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099264    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-systemd-units\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-etc-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-cni-bin\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovn-node-metrics-cert\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-registration-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cni-binary-copy\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-etc-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d565593-7fbb-41ef-8856-8b6dfa88a62c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-device-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-sys-fs\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.099657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-device-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099300    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-slash\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-env-overrides\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b96da89-37da-4ea3-a3f6-33282f05e059-host\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovnkube-config\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-sys-fs\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-cni-bin\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b96da89-37da-4ea3-a3f6-33282f05e059-host\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b96da89-37da-4ea3-a3f6-33282f05e059-serviceca\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-run-netns\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-cni-netd\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.099704    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-var-lib-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovnkube-script-lib\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e353f605-9075-414d-aa37-b8b53f094c02-dbus\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-registration-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.100521 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-node-log\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-var-lib-openvswitch\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.099998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-run-ovn-kubernetes\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-socket-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-ovn\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-cni-netd\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-run-netns\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.100136    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret podName:e353f605-9075-414d-aa37-b8b53f094c02 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:56.60009631 +0000 UTC m=+6.114178440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret") pod "global-pull-secret-syncer-g698z" (UID: "e353f605-9075-414d-aa37-b8b53f094c02") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovnkube-config\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cni-binary-copy\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100194    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-node-log\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-run-ovn-kubernetes\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-log-socket\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-ovn\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b96da89-37da-4ea3-a3f6-33282f05e059-serviceca\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:56.101608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-socket-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e353f605-9075-414d-aa37-b8b53f094c02-dbus\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-log-socket\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r2hv\" (UniqueName: \"kubernetes.io/projected/b6f86368-5761-4e43-bb5b-022524a4d82c-kube-api-access-7r2hv\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28c8l\" (UniqueName: \"kubernetes.io/projected/2a30635a-06fa-47d4-9416-be835f7efd0a-kube-api-access-28c8l\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-systemd\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxgm\" (UniqueName: \"kubernetes.io/projected/7d565593-7fbb-41ef-8856-8b6dfa88a62c-kube-api-access-jjxgm\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-etc-selinux\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a30635a-06fa-47d4-9416-be835f7efd0a-iptables-alerter-script\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7fca20-9b25-4a26-a49d-3089acb2966e-run-systemd\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovnkube-script-lib\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d565593-7fbb-41ef-8856-8b6dfa88a62c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.100729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6f86368-5761-4e43-bb5b-022524a4d82c-etc-selinux\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.102428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.101000    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a30635a-06fa-47d4-9416-be835f7efd0a-iptables-alerter-script\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:56.103120 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.103054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7fca20-9b25-4a26-a49d-3089acb2966e-ovn-node-metrics-cert\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.108905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.108882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnzm\" (UniqueName: \"kubernetes.io/projected/6b7fca20-9b25-4a26-a49d-3089acb2966e-kube-api-access-9nnzm\") pod \"ovnkube-node-94lfn\" (UID: \"6b7fca20-9b25-4a26-a49d-3089acb2966e\") " pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.109018 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.108954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r2hv\" (UniqueName: \"kubernetes.io/projected/b6f86368-5761-4e43-bb5b-022524a4d82c-kube-api-access-7r2hv\") pod \"aws-ebs-csi-driver-node-x4pzm\" (UID: \"b6f86368-5761-4e43-bb5b-022524a4d82c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.109193 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.109177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxgm\" (UniqueName: \"kubernetes.io/projected/7d565593-7fbb-41ef-8856-8b6dfa88a62c-kube-api-access-jjxgm\") pod \"multus-additional-cni-plugins-9w82s\" (UID: \"7d565593-7fbb-41ef-8856-8b6dfa88a62c\") " pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.109284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.109267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h85s\" (UniqueName: \"kubernetes.io/projected/9b96da89-37da-4ea3-a3f6-33282f05e059-kube-api-access-2h85s\") pod \"node-ca-2p8lk\" (UID: \"9b96da89-37da-4ea3-a3f6-33282f05e059\") " pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:56.109636 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.109620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28c8l\" (UniqueName: \"kubernetes.io/projected/2a30635a-06fa-47d4-9416-be835f7efd0a-kube-api-access-28c8l\") pod \"iptables-alerter-k774j\" (UID: \"2a30635a-06fa-47d4-9416-be835f7efd0a\") " pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:56.172992 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.172953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:05:56.177179 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.177157    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-28jf7"
Sep 09 03:05:56.184079 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.184057    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9w82s"
Sep 09 03:05:56.185191 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:56.185160    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fae6eb0_381c_4cc5_9098_8c95abeca3ba.slice/crio-b79975ca0656a7d303a9d08290d8be77575f60c035b39f23117f3a7f2a91deeb WatchSource:0}: Error finding container b79975ca0656a7d303a9d08290d8be77575f60c035b39f23117f3a7f2a91deeb: Status 404 returned error can't find the container with id b79975ca0656a7d303a9d08290d8be77575f60c035b39f23117f3a7f2a91deeb
Sep 09 03:05:56.187256 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:56.187138    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3c44b2_531b_4a24_8daf_a05142141da9.slice/crio-9b129c16d0e4743fc80ad5c3d5d288d1c4d6d908357f5c021c5be9f5066bfc26 WatchSource:0}: Error finding container 9b129c16d0e4743fc80ad5c3d5d288d1c4d6d908357f5c021c5be9f5066bfc26: Status 404 returned error can't find the container with id 9b129c16d0e4743fc80ad5c3d5d288d1c4d6d908357f5c021c5be9f5066bfc26
Sep 09 03:05:56.187453 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.187432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9zhgm"
Sep 09 03:05:56.192791 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.192771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bccdk"
Sep 09 03:05:56.195891 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:56.195869    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d565593_7fbb_41ef_8856_8b6dfa88a62c.slice/crio-58c28324a1ddeafc3cbbc5469385fc9b79a2121e1778389823e8e694ffd2f8fd WatchSource:0}: Error finding container 58c28324a1ddeafc3cbbc5469385fc9b79a2121e1778389823e8e694ffd2f8fd: Status 404 returned error can't find the container with id 58c28324a1ddeafc3cbbc5469385fc9b79a2121e1778389823e8e694ffd2f8fd
Sep 09 03:05:56.200672 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.200594    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:05:56.203391 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:56.203368    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f5888a_646d_4f07_9073_aebd7f316182.slice/crio-8a1959bb13004d622430a1fb6efe6d2d35a3599c172afde7a35ee860ba1f4c25 WatchSource:0}: Error finding container 8a1959bb13004d622430a1fb6efe6d2d35a3599c172afde7a35ee860ba1f4c25: Status 404 returned error can't find the container with id 8a1959bb13004d622430a1fb6efe6d2d35a3599c172afde7a35ee860ba1f4c25
Sep 09 03:05:56.204330 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.204314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm"
Sep 09 03:05:56.209808 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.209790    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2p8lk"
Sep 09 03:05:56.210553 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:56.210534    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7fca20_9b25_4a26_a49d_3089acb2966e.slice/crio-42d9b475bbe80131e0a78c22a3d96580603a1c36b26bb07b02747f6dcf30613e WatchSource:0}: Error finding container 42d9b475bbe80131e0a78c22a3d96580603a1c36b26bb07b02747f6dcf30613e: Status 404 returned error can't find the container with id 42d9b475bbe80131e0a78c22a3d96580603a1c36b26bb07b02747f6dcf30613e
Sep 09 03:05:56.213560 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:56.213539    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f86368_5761_4e43_bb5b_022524a4d82c.slice/crio-a2abde4b13a1e57d4b8dd5a7cc57b6cf88bceeec1fd4a36fecd4dc78065d49d1 WatchSource:0}: Error finding container a2abde4b13a1e57d4b8dd5a7cc57b6cf88bceeec1fd4a36fecd4dc78065d49d1: Status 404 returned error can't find the container with id a2abde4b13a1e57d4b8dd5a7cc57b6cf88bceeec1fd4a36fecd4dc78065d49d1
Sep 09 03:05:56.213658 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.213571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k774j"
Sep 09 03:05:56.220233 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:56.220206    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b96da89_37da_4ea3_a3f6_33282f05e059.slice/crio-a4accf0f1e04e4890d02c7e9ab80985f776bfba36601e0edfecec9a0f888675d WatchSource:0}: Error finding container a4accf0f1e04e4890d02c7e9ab80985f776bfba36601e0edfecec9a0f888675d: Status 404 returned error can't find the container with id a4accf0f1e04e4890d02c7e9ab80985f776bfba36601e0edfecec9a0f888675d
Sep 09 03:05:56.223184 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:05:56.223156    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a30635a_06fa_47d4_9416_be835f7efd0a.slice/crio-980daca19e7a4ea9dba889e9d1710a142c3761f1b60188d74a6d129d3cdf496f WatchSource:0}: Error finding container 980daca19e7a4ea9dba889e9d1710a142c3761f1b60188d74a6d129d3cdf496f: Status 404 returned error can't find the container with id 980daca19e7a4ea9dba889e9d1710a142c3761f1b60188d74a6d129d3cdf496f
Sep 09 03:05:56.505228 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.505120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:05:56.505228 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.505200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:56.505475 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.505310    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:56.505475 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.505336    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:56.505475 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.505361    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:56.505475 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.505375    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dnwqj for pod openshift-network-diagnostics/network-check-target-nkqd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:56.505475 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.505379    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:57.505359726 +0000 UTC m=+7.019441875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:56.505475 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.505414    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj podName:bafde9e9-c0d3-4b04-b0eb-d77c722be9ec nodeName:}" failed. No retries permitted until 2026-09-09 03:05:57.505402556 +0000 UTC m=+7.019484686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnwqj" (UniqueName: "kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj") pod "network-check-target-nkqd9" (UID: "bafde9e9-c0d3-4b04-b0eb-d77c722be9ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:56.606567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.606507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:56.606766 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.606703    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:56.606833 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:56.606824    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret podName:e353f605-9075-414d-aa37-b8b53f094c02 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:57.60680209 +0000 UTC m=+7.120884221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret") pod "global-pull-secret-syncer-g698z" (UID: "e353f605-9075-414d-aa37-b8b53f094c02") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:56.900952 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.900791    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:00:56 +0000 UTC" deadline="2028-03-11 12:50:06.111730623 +0000 UTC"
Sep 09 03:05:56.900952 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:56.900882    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13185h44m9.210853489s"
Sep 09 03:05:57.004596 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.004559    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:57.004759 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.004714    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:05:57.034554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.034464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k774j" event={"ID":"2a30635a-06fa-47d4-9416-be835f7efd0a","Type":"ContainerStarted","Data":"980daca19e7a4ea9dba889e9d1710a142c3761f1b60188d74a6d129d3cdf496f"}
Sep 09 03:05:57.041471 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.041432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"42d9b475bbe80131e0a78c22a3d96580603a1c36b26bb07b02747f6dcf30613e"}
Sep 09 03:05:57.047525 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.047493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9zhgm" event={"ID":"742fbb48-0057-4d26-9b3c-2493b94a952e","Type":"ContainerStarted","Data":"2796eb5925f1c5e188808d343a8a1d85dec66845a2aa5efc66c77eecfe5ab8c3"}
Sep 09 03:05:57.051740 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.051710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w82s" event={"ID":"7d565593-7fbb-41ef-8856-8b6dfa88a62c","Type":"ContainerStarted","Data":"58c28324a1ddeafc3cbbc5469385fc9b79a2121e1778389823e8e694ffd2f8fd"}
Sep 09 03:05:57.054813 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.054781    2583 generic.go:356] "Generic (PLEG): container finished" podID="aecef6bfb645f61d4d21f56287e6cd97" containerID="ea2a3fa30252f2e732178f5af476737261198d2aea26561c6892e33ace63e2a1" exitCode=0
Sep 09 03:05:57.054928 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.054873    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal" event={"ID":"aecef6bfb645f61d4d21f56287e6cd97","Type":"ContainerDied","Data":"ea2a3fa30252f2e732178f5af476737261198d2aea26561c6892e33ace63e2a1"}
Sep 09 03:05:57.062165 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.062098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2p8lk" event={"ID":"9b96da89-37da-4ea3-a3f6-33282f05e059","Type":"ContainerStarted","Data":"a4accf0f1e04e4890d02c7e9ab80985f776bfba36601e0edfecec9a0f888675d"}
Sep 09 03:05:57.067220 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.067107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm" event={"ID":"b6f86368-5761-4e43-bb5b-022524a4d82c","Type":"ContainerStarted","Data":"a2abde4b13a1e57d4b8dd5a7cc57b6cf88bceeec1fd4a36fecd4dc78065d49d1"}
Sep 09 03:05:57.075047 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.074722    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bccdk" event={"ID":"39f5888a-646d-4f07-9073-aebd7f316182","Type":"ContainerStarted","Data":"8a1959bb13004d622430a1fb6efe6d2d35a3599c172afde7a35ee860ba1f4c25"}
Sep 09 03:05:57.080191 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.080166    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-28jf7" event={"ID":"0c3c44b2-531b-4a24-8daf-a05142141da9","Type":"ContainerStarted","Data":"9b129c16d0e4743fc80ad5c3d5d288d1c4d6d908357f5c021c5be9f5066bfc26"}
Sep 09 03:05:57.084980 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.084956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x4mml" event={"ID":"4fae6eb0-381c-4cc5-9098-8c95abeca3ba","Type":"ContainerStarted","Data":"b79975ca0656a7d303a9d08290d8be77575f60c035b39f23117f3a7f2a91deeb"}
Sep 09 03:05:57.516951 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.516594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:05:57.517155 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.516976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:57.517155 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.517148    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:57.517287 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.517219    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:59.517197991 +0000 UTC m=+9.031280134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:57.517747 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.517726    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:57.517865 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.517753    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:57.517865 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.517768    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dnwqj for pod openshift-network-diagnostics/network-check-target-nkqd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:57.517865 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.517831    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj podName:bafde9e9-c0d3-4b04-b0eb-d77c722be9ec nodeName:}" failed. No retries permitted until 2026-09-09 03:05:59.517812746 +0000 UTC m=+9.031894884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnwqj" (UniqueName: "kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj") pod "network-check-target-nkqd9" (UID: "bafde9e9-c0d3-4b04-b0eb-d77c722be9ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:57.617697 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.617652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:57.617943 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.617882    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:57.618028 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:57.617955    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret podName:e353f605-9075-414d-aa37-b8b53f094c02 nodeName:}" failed. No retries permitted until 2026-09-09 03:05:59.617931929 +0000 UTC m=+9.132014059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret") pod "global-pull-secret-syncer-g698z" (UID: "e353f605-9075-414d-aa37-b8b53f094c02") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:57.901767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.901672    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:00:56 +0000 UTC" deadline="2028-03-17 03:00:48.773010159 +0000 UTC"
Sep 09 03:05:57.901767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:57.901718    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13319h54m50.871296874s"
Sep 09 03:05:58.001914 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:58.001871    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:58.002110 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:58.002020    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:05:58.002464 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:58.002442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:05:58.002597 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:58.002541    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:05:58.105455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:58.105380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal" event={"ID":"aecef6bfb645f61d4d21f56287e6cd97","Type":"ContainerStarted","Data":"0e1d16c54857c3527c4b606810a1131c95b1610b6be237f9123b4bef36a8761f"}
Sep 09 03:05:58.134928 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:58.132194    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-233.ec2.internal" podStartSLOduration=3.132168455 podStartE2EDuration="3.132168455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:05:58.131329017 +0000 UTC m=+7.645411172" watchObservedRunningTime="2026-09-09 03:05:58.132168455 +0000 UTC m=+7.646250610"
Sep 09 03:05:59.002174 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:59.001956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:59.002174 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.002097    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:05:59.548678 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:59.548634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:05:59.548881 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:59.548724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:05:59.548881 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.548854    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:05:59.548881 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.548873    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:59.548881 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.548879    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:05:59.549096 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.548893    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dnwqj for pod openshift-network-diagnostics/network-check-target-nkqd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:59.549096 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.548942    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:03.548922189 +0000 UTC m=+13.063004318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:05:59.549096 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.548962    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj podName:bafde9e9-c0d3-4b04-b0eb-d77c722be9ec nodeName:}" failed. No retries permitted until 2026-09-09 03:06:03.548952934 +0000 UTC m=+13.063035063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnwqj" (UniqueName: "kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj") pod "network-check-target-nkqd9" (UID: "bafde9e9-c0d3-4b04-b0eb-d77c722be9ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:05:59.649977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:05:59.649401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:05:59.649977 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.649566    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:05:59.649977 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:05:59.649633    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret podName:e353f605-9075-414d-aa37-b8b53f094c02 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:03.649614628 +0000 UTC m=+13.163696760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret") pod "global-pull-secret-syncer-g698z" (UID: "e353f605-9075-414d-aa37-b8b53f094c02") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:00.001898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:00.001329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:00.001898 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:00.001454    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:00.001898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:00.001758    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:00.001898 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:00.001846    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:01.002654 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:01.002215    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:01.002654 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:01.002323    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:02.001384 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:02.001332    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:02.001553 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:02.001481    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:02.002116 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:02.002093    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:02.002259 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:02.002205    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:03.002060 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:03.002023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:03.002467 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.002172    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:03.583331 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:03.582486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:03.583331 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:03.582572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:03.583331 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.582696    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:03.583331 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.582760    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:11.582739403 +0000 UTC m=+21.096821555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:03.583331 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.583211    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:03.583331 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.583229    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:03.583331 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.583241    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dnwqj for pod openshift-network-diagnostics/network-check-target-nkqd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:03.583331 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.583284    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj podName:bafde9e9-c0d3-4b04-b0eb-d77c722be9ec nodeName:}" failed. No retries permitted until 2026-09-09 03:06:11.583270121 +0000 UTC m=+21.097352264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnwqj" (UniqueName: "kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj") pod "network-check-target-nkqd9" (UID: "bafde9e9-c0d3-4b04-b0eb-d77c722be9ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:03.683613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:03.683576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:03.683811 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.683735    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:03.683910 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:03.683841    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret podName:e353f605-9075-414d-aa37-b8b53f094c02 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:11.683802434 +0000 UTC m=+21.197884586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret") pod "global-pull-secret-syncer-g698z" (UID: "e353f605-9075-414d-aa37-b8b53f094c02") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:04.000986 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:04.000910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:04.001150 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:04.001061    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:04.001150 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:04.000910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:04.001269 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:04.001194    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:05.002022 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:05.001987    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:05.002458 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:05.002135    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:06.001755 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:06.001714    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:06.001932 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:06.001881    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:06.002011 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:06.001937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:06.002053 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:06.002031    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:07.001048 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:07.001015    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:07.001216 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:07.001138    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:08.001163 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:08.001119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:08.001163 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:08.001159    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:08.001696 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:08.001258    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:08.001696 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:08.001380    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:09.001251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:09.001217    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:09.001680 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:09.001333    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:10.001768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:10.001733    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:10.002217 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:10.001785    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:10.002217 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:10.001911    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:10.002217 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:10.002046    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:11.001865 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:11.001816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:11.002336 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.001947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:11.644083 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:11.644043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:11.644278 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:11.644137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:11.644278 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.644231    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:11.644278 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.644252    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:11.644278 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.644266    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dnwqj for pod openshift-network-diagnostics/network-check-target-nkqd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:11.644458 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.644285    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:11.644458 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.644324    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj podName:bafde9e9-c0d3-4b04-b0eb-d77c722be9ec nodeName:}" failed. No retries permitted until 2026-09-09 03:06:27.644308931 +0000 UTC m=+37.158391060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnwqj" (UniqueName: "kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj") pod "network-check-target-nkqd9" (UID: "bafde9e9-c0d3-4b04-b0eb-d77c722be9ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:11.644458 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.644346    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:27.644333599 +0000 UTC m=+37.158415726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:11.745078 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:11.745040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:11.745277 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.745198    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:11.745277 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:11.745258    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret podName:e353f605-9075-414d-aa37-b8b53f094c02 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:27.745238725 +0000 UTC m=+37.259320852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret") pod "global-pull-secret-syncer-g698z" (UID: "e353f605-9075-414d-aa37-b8b53f094c02") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:12.001503 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:12.001417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:12.001672 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:12.001426    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:12.001672 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:12.001526    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:12.001672 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:12.001635    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:13.000964 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:13.000933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:13.001514 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:13.001061    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:14.001030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:14.000968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:14.001424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:14.000967    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:14.001424 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:14.001102    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:14.001424 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:14.001140    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:15.001717 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.001528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:15.002361 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:15.001823    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:15.093253 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.093103    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 03:06:15.151442 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.151410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2p8lk" event={"ID":"9b96da89-37da-4ea3-a3f6-33282f05e059","Type":"ContainerStarted","Data":"4fd1693b6c1511e7313e5ea125b2778da3013cd5825ce2b691ca9fc0ebe16e37"}
Sep 09 03:06:15.153336 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.153311    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm" event={"ID":"b6f86368-5761-4e43-bb5b-022524a4d82c","Type":"ContainerStarted","Data":"0c6b89a8a594603ca2ddeb6673408fb9c90a6fdbbee92cb753e31c218c2baa18"}
Sep 09 03:06:15.153454 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.153365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm" event={"ID":"b6f86368-5761-4e43-bb5b-022524a4d82c","Type":"ContainerStarted","Data":"3e77f37f2cdc5736157187f7f865fa6057fca757803f57ad6aab580f34c4d404"}
Sep 09 03:06:15.154939 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.154919    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bccdk" event={"ID":"39f5888a-646d-4f07-9073-aebd7f316182","Type":"ContainerStarted","Data":"e032916a0db54cb5c0e757e4133345576bdb63d1c5bc7b446bb1bf4b651b36e3"}
Sep 09 03:06:15.156425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.156403    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-28jf7" event={"ID":"0c3c44b2-531b-4a24-8daf-a05142141da9","Type":"ContainerStarted","Data":"ca10c2073c121e1e4a70361530f353e0f5d728454c54be788d677c6d6f4cf2d4"}
Sep 09 03:06:15.160771 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.160745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x4mml" event={"ID":"4fae6eb0-381c-4cc5-9098-8c95abeca3ba","Type":"ContainerStarted","Data":"01519742b3f45aa8a164abe10986f4dcf83ff7cc457a6bf2513b328c26dffb7d"}
Sep 09 03:06:15.163613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.163596    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:06:15.163976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.163953    2583 generic.go:356] "Generic (PLEG): container finished" podID="6b7fca20-9b25-4a26-a49d-3089acb2966e" containerID="032a3d72bc6f8e0899e5465ae60ec93e7bb0262351397f7c1d2216b353895576" exitCode=1
Sep 09 03:06:15.164043 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.164019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"368365aff541ecf0145109e470982b34b086693aaf41777a2e67bc40a715b3c6"}
Sep 09 03:06:15.164123 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.164100    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"a41d560c3478179dcf814d09a380526971f1ff00b505e9331070b181cb76139e"}
Sep 09 03:06:15.164251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.164151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"ba64c1148e6b8d0910a6dc3514a8ab8608f76fd3921953fbd28703e9be4b7aaf"}
Sep 09 03:06:15.164251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.164196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"adebbad2b1f4bd35cc8c673547a033b745a9188ab2c73fc2988a94e2a5a4c096"}
Sep 09 03:06:15.164251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.164241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerDied","Data":"032a3d72bc6f8e0899e5465ae60ec93e7bb0262351397f7c1d2216b353895576"}
Sep 09 03:06:15.164387 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.164288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"5e8181f108694afeddf95106a7d4bc15bf6ae7a4f40d07b49249ff1b5fb9f0a5"}
Sep 09 03:06:15.164665 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.164633    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2p8lk" podStartSLOduration=6.460526003 podStartE2EDuration="24.164621116s" totalImagesPullingTime="17.704095113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:15.163959324 +0000 UTC m=+24.678041497" watchObservedRunningTime="2026-09-09 03:06:15.164621116 +0000 UTC m=+24.678703265"
Sep 09 03:06:15.165441 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.165421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9zhgm" event={"ID":"742fbb48-0057-4d26-9b3c-2493b94a952e","Type":"ContainerStarted","Data":"a39fa99117c906d7138f510c436111afa519bbe071347d5a8fb605bef12927b5"}
Sep 09 03:06:15.167019 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.166989    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d565593-7fbb-41ef-8856-8b6dfa88a62c" containerID="6cb9afde7de8b6ed68b7ac2cba9bef897d20c07ebc1af43835070b1309721a35" exitCode=0
Sep 09 03:06:15.167098 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.167031    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w82s" event={"ID":"7d565593-7fbb-41ef-8856-8b6dfa88a62c","Type":"ContainerDied","Data":"6cb9afde7de8b6ed68b7ac2cba9bef897d20c07ebc1af43835070b1309721a35"}
Sep 09 03:06:15.179587 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.179551    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-28jf7" podStartSLOduration=6.340983083 podStartE2EDuration="24.179538117s" totalImagesPullingTime="17.838555034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:15.179244084 +0000 UTC m=+24.693326253" watchObservedRunningTime="2026-09-09 03:06:15.179538117 +0000 UTC m=+24.693620294"
Sep 09 03:06:15.209678 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.209638    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bccdk" podStartSLOduration=6.33778113 podStartE2EDuration="24.209623775s" totalImagesPullingTime="17.871842645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:15.209420139 +0000 UTC m=+24.723502299" watchObservedRunningTime="2026-09-09 03:06:15.209623775 +0000 UTC m=+24.723705924"
Sep 09 03:06:15.209998 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.209976    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x4mml" podStartSLOduration=10.57053487 podStartE2EDuration="24.209966977s" totalImagesPullingTime="13.639432107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:15.193749216 +0000 UTC m=+24.707831366" watchObservedRunningTime="2026-09-09 03:06:15.209966977 +0000 UTC m=+24.724049123"
Sep 09 03:06:15.242290 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.242252    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9zhgm" podStartSLOduration=6.5171201199999995 podStartE2EDuration="24.242238304s" totalImagesPullingTime="17.725118184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:15.242108858 +0000 UTC m=+24.756191007" watchObservedRunningTime="2026-09-09 03:06:15.242238304 +0000 UTC m=+24.756320453"
Sep 09 03:06:15.934769 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.934659    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T03:06:15.093250523Z","UUID":"1a78da40-cd3c-4bb5-99c9-4693d21ef9df","Handler":null,"Name":"","Endpoint":""}
Sep 09 03:06:15.936666 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.936641    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 03:06:15.936798 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:15.936678    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 03:06:16.001166 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:16.001134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:16.001331 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:16.001176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:16.001331 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:16.001265    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:16.001455 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:16.001339    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:16.172753 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:16.172582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k774j" event={"ID":"2a30635a-06fa-47d4-9416-be835f7efd0a","Type":"ContainerStarted","Data":"e7c799d0d3462e12ee1691b80c0f27f6a60e939a14a31489d19b3f0fd7cb300f"}
Sep 09 03:06:16.173241 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:16.173202    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:06:16.173704 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:16.173681    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:06:16.187375 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:16.187283    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k774j" podStartSLOduration=7.386836285 podStartE2EDuration="25.187266553s" totalImagesPullingTime="17.800430268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:16.187082938 +0000 UTC m=+25.701165127" watchObservedRunningTime="2026-09-09 03:06:16.187266553 +0000 UTC m=+25.701348712"
Sep 09 03:06:17.001803 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:17.001769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:17.001993 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:17.001927    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:17.178095 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:17.178041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm" event={"ID":"b6f86368-5761-4e43-bb5b-022524a4d82c","Type":"ContainerStarted","Data":"13330aae518636417046091aa0ebca103b4cdb616d8f52a63b3c4f3ca2751f0e"}
Sep 09 03:06:17.182294 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:17.182256    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:06:17.182781 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:17.182749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"f74b4ed82b5546831ddaed7212dbf674e2de2867bf119a8daa89d14705ffa9b8"}
Sep 09 03:06:17.195134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:17.195073    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4pzm" podStartSLOduration=6.366397674 podStartE2EDuration="26.195052881s" totalImagesPullingTime="19.828655207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:17.194581711 +0000 UTC m=+26.708663905" watchObservedRunningTime="2026-09-09 03:06:17.195052881 +0000 UTC m=+26.709135032"
Sep 09 03:06:17.560264 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:17.560177    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:06:17.560732 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:17.560708    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x4mml"
Sep 09 03:06:18.001172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:18.001145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:18.001378 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:18.001145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:18.001378 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:18.001264    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:18.001480 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:18.001373    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:19.000952 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:19.000918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:19.001387 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:19.001041    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:19.192126 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:19.191953    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:06:19.192494 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:19.192469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"e1f2df073b84aee70781e922a24d7f4f03be70235eff4898b560e8cd8e821cef"}
Sep 09 03:06:19.193433 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:19.193305    2583 scope.go:117] "RemoveContainer" containerID="032a3d72bc6f8e0899e5465ae60ec93e7bb0262351397f7c1d2216b353895576"
Sep 09 03:06:20.001568 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.001369    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:20.002164 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.001417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:20.002164 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:20.001678    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:20.002164 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:20.001707    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:20.198861 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.198812    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:06:20.199217 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.199196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" event={"ID":"6b7fca20-9b25-4a26-a49d-3089acb2966e","Type":"ContainerStarted","Data":"e4b6183e21ff4408519eebe5c4509e18ef6e143caea3310a68f58e2dc40aa530"}
Sep 09 03:06:20.199264 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.199256    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 03:06:20.199492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.199471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:06:20.199613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.199533    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:06:20.201479 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.201456    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d565593-7fbb-41ef-8856-8b6dfa88a62c" containerID="bd07e80966f19af08302370393efe582e894846ae168def5d3f2113f163039cd" exitCode=0
Sep 09 03:06:20.201588 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.201492    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w82s" event={"ID":"7d565593-7fbb-41ef-8856-8b6dfa88a62c","Type":"ContainerDied","Data":"bd07e80966f19af08302370393efe582e894846ae168def5d3f2113f163039cd"}
Sep 09 03:06:20.215624 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.215602    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:06:20.215733 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.215700    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:06:20.226203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:20.226164    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn" podStartSLOduration=11.395003609 podStartE2EDuration="29.226152486s" totalImagesPullingTime="17.831148877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:20.224941706 +0000 UTC m=+29.739023876" watchObservedRunningTime="2026-09-09 03:06:20.226152486 +0000 UTC m=+29.740234635"
Sep 09 03:06:21.005277 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.002940    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:21.005277 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:21.003917    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:21.033668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.033642    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g698z"]
Sep 09 03:06:21.036942 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.036910    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nkqd9"]
Sep 09 03:06:21.037086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.037039    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:21.037170 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:21.037152    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:21.037664 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.037515    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zqzm9"]
Sep 09 03:06:21.037664 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.037621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:21.037866 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:21.037712    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:21.198253 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.198222    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:06:21.206381 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.206349    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d565593-7fbb-41ef-8856-8b6dfa88a62c" containerID="52ef7a0dbc3e99532a9482fc0e57ed9612f1ea81078790372e1e76a26935980d" exitCode=0
Sep 09 03:06:21.206504 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.206379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w82s" event={"ID":"7d565593-7fbb-41ef-8856-8b6dfa88a62c","Type":"ContainerDied","Data":"52ef7a0dbc3e99532a9482fc0e57ed9612f1ea81078790372e1e76a26935980d"}
Sep 09 03:06:21.206504 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:21.206487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:21.206601 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:21.206581    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:22.211756 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:22.211715    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d565593-7fbb-41ef-8856-8b6dfa88a62c" containerID="5cfb2552c665e353da33a63ea392a715e4ab09196790803f2f3232f50a0cae90" exitCode=0
Sep 09 03:06:22.212231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:22.211784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w82s" event={"ID":"7d565593-7fbb-41ef-8856-8b6dfa88a62c","Type":"ContainerDied","Data":"5cfb2552c665e353da33a63ea392a715e4ab09196790803f2f3232f50a0cae90"}
Sep 09 03:06:23.001574 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:23.001539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:23.001780 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:23.001582    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:23.001780 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:23.001546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:23.001780 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:23.001682    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:23.001780 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:23.001749    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:23.001986 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:23.001841    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:25.001198 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:25.001164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:25.001870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:25.001171    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:25.001870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:25.001299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:25.001870 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:25.001302    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:25.001870 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:25.001381    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:25.001870 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:25.001481    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:27.001228 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.001200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:27.001589 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.001308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:27.001589 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.001316    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:06:27.001589 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.001391    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g698z" podUID="e353f605-9075-414d-aa37-b8b53f094c02"
Sep 09 03:06:27.001589 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.001411    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:27.001589 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.001490    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nkqd9" podUID="bafde9e9-c0d3-4b04-b0eb-d77c722be9ec"
Sep 09 03:06:27.227165 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.227069    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d565593-7fbb-41ef-8856-8b6dfa88a62c" containerID="c595da78efe7a401740fb6d884adac303cc302307abc772ef99d9bde3658d2c6" exitCode=0
Sep 09 03:06:27.227165 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.227109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w82s" event={"ID":"7d565593-7fbb-41ef-8856-8b6dfa88a62c","Type":"ContainerDied","Data":"c595da78efe7a401740fb6d884adac303cc302307abc772ef99d9bde3658d2c6"}
Sep 09 03:06:27.247014 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.246889    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-233.ec2.internal" event="NodeReady"
Sep 09 03:06:27.247206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.247033    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 03:06:27.286549 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.286523    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rg75s"]
Sep 09 03:06:27.316940 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.316915    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dvljc"]
Sep 09 03:06:27.317065 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.317037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.319374 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.319348    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8s7tp\""
Sep 09 03:06:27.319481 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.319350    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 03:06:27.319481 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.319385    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 03:06:27.333331 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.333312    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rg75s"]
Sep 09 03:06:27.333431 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.333343    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dvljc"]
Sep 09 03:06:27.333431 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.333423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dvljc"
Sep 09 03:06:27.335550 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.335533    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 03:06:27.335661 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.335591    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 03:06:27.335971 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.335957    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fjjq4\""
Sep 09 03:06:27.336036 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.335978    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 03:06:27.457137 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.457095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqs4\" (UniqueName: \"kubernetes.io/projected/dae8f832-f923-4354-9649-b1ddc523ebf8-kube-api-access-mlqs4\") pod \"ingress-canary-dvljc\" (UID: \"dae8f832-f923-4354-9649-b1ddc523ebf8\") " pod="openshift-ingress-canary/ingress-canary-dvljc"
Sep 09 03:06:27.457137 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.457136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dae8f832-f923-4354-9649-b1ddc523ebf8-cert\") pod \"ingress-canary-dvljc\" (UID: \"dae8f832-f923-4354-9649-b1ddc523ebf8\") " pod="openshift-ingress-canary/ingress-canary-dvljc"
Sep 09 03:06:27.457334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.457165    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-tmp-dir\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.457334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.457235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-config-volume\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.457334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.457258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.457334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.457274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4kkh\" (UniqueName: \"kubernetes.io/projected/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-kube-api-access-g4kkh\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.558495 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.558420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqs4\" (UniqueName: \"kubernetes.io/projected/dae8f832-f923-4354-9649-b1ddc523ebf8-kube-api-access-mlqs4\") pod \"ingress-canary-dvljc\" (UID: \"dae8f832-f923-4354-9649-b1ddc523ebf8\") " pod="openshift-ingress-canary/ingress-canary-dvljc"
Sep 09 03:06:27.558495 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.558454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dae8f832-f923-4354-9649-b1ddc523ebf8-cert\") pod \"ingress-canary-dvljc\" (UID: \"dae8f832-f923-4354-9649-b1ddc523ebf8\") " pod="openshift-ingress-canary/ingress-canary-dvljc"
Sep 09 03:06:27.558495 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.558491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-tmp-dir\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.558738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.558509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-config-volume\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.558738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.558535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.558738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.558554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4kkh\" (UniqueName: \"kubernetes.io/projected/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-kube-api-access-g4kkh\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.558738 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.558704    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:27.558957 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.558781    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:28.058760448 +0000 UTC m=+37.572842593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:27.559078 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.558881    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-tmp-dir\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.559257 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.559231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-config-volume\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.562734 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.562713    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dae8f832-f923-4354-9649-b1ddc523ebf8-cert\") pod \"ingress-canary-dvljc\" (UID: \"dae8f832-f923-4354-9649-b1ddc523ebf8\") " pod="openshift-ingress-canary/ingress-canary-dvljc"
Sep 09 03:06:27.567542 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.567512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4kkh\" (UniqueName: \"kubernetes.io/projected/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-kube-api-access-g4kkh\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:27.567694 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.567679    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqs4\" (UniqueName: \"kubernetes.io/projected/dae8f832-f923-4354-9649-b1ddc523ebf8-kube-api-access-mlqs4\") pod \"ingress-canary-dvljc\" (UID: \"dae8f832-f923-4354-9649-b1ddc523ebf8\") " pod="openshift-ingress-canary/ingress-canary-dvljc"
Sep 09 03:06:27.641693 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.641661    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dvljc"
Sep 09 03:06:27.659979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.659951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:27.660043 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.660005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:27.660105 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.660093    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:27.660139 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.660116    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:06:27.660171 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.660134    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:06:27.660171 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.660155    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:59.660141401 +0000 UTC m=+69.174223529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:06:27.660171 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.660156    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dnwqj for pod openshift-network-diagnostics/network-check-target-nkqd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:27.660266 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.660196    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj podName:bafde9e9-c0d3-4b04-b0eb-d77c722be9ec nodeName:}" failed. No retries permitted until 2026-09-09 03:06:59.660185073 +0000 UTC m=+69.174267201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnwqj" (UniqueName: "kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj") pod "network-check-target-nkqd9" (UID: "bafde9e9-c0d3-4b04-b0eb-d77c722be9ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:06:27.760914 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.760876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:27.761071 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.761017    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:27.761112 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:27.761085    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret podName:e353f605-9075-414d-aa37-b8b53f094c02 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:59.761070291 +0000 UTC m=+69.275152419 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret") pod "global-pull-secret-syncer-g698z" (UID: "e353f605-9075-414d-aa37-b8b53f094c02") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:06:27.992026 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:27.991996    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dvljc"]
Sep 09 03:06:27.995616 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:06:27.995585    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae8f832_f923_4354_9649_b1ddc523ebf8.slice/crio-5d533589593573dace21460129ec005c4b811e85fb91e987a8d6fb5f79bbe60f WatchSource:0}: Error finding container 5d533589593573dace21460129ec005c4b811e85fb91e987a8d6fb5f79bbe60f: Status 404 returned error can't find the container with id 5d533589593573dace21460129ec005c4b811e85fb91e987a8d6fb5f79bbe60f
Sep 09 03:06:28.062848 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:28.062800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:28.063540 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:28.062979    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:28.063540 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:28.063059    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:29.063039036 +0000 UTC m=+38.577121164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:28.230649 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:28.230611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dvljc" event={"ID":"dae8f832-f923-4354-9649-b1ddc523ebf8","Type":"ContainerStarted","Data":"5d533589593573dace21460129ec005c4b811e85fb91e987a8d6fb5f79bbe60f"}
Sep 09 03:06:28.233954 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:28.233922    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d565593-7fbb-41ef-8856-8b6dfa88a62c" containerID="d1d77e6021a25862de6308b3c55be34e92c95493a7da54a05633ceb44fffe3fc" exitCode=0
Sep 09 03:06:28.234111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:28.233964    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w82s" event={"ID":"7d565593-7fbb-41ef-8856-8b6dfa88a62c","Type":"ContainerDied","Data":"d1d77e6021a25862de6308b3c55be34e92c95493a7da54a05633ceb44fffe3fc"}
Sep 09 03:06:29.001742 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.001701    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:29.001944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.001701    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:29.001944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.001885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:29.004435 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.004399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bzb6q\""
Sep 09 03:06:29.004567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.004547    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:06:29.005880 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.005560    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:06:29.005880 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.005594    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qqmk2\""
Sep 09 03:06:29.005880 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.005605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:06:29.005880 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.005569    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:06:29.070176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.070144    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:29.070662 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:29.070299    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:29.070662 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:29.070372    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:31.070351295 +0000 UTC m=+40.584433427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:29.241409 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.241375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w82s" event={"ID":"7d565593-7fbb-41ef-8856-8b6dfa88a62c","Type":"ContainerStarted","Data":"d33751b39f383979f776d484d26e88147276bf0a39dd3d1b57fe07468f5419d4"}
Sep 09 03:06:29.261116 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:29.261022    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9w82s" podStartSLOduration=10.272105944 podStartE2EDuration="38.261003312s" totalImagesPullingTime="27.988897368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:29.259747478 +0000 UTC m=+38.773829627" watchObservedRunningTime="2026-09-09 03:06:29.261003312 +0000 UTC m=+38.775085464"
Sep 09 03:06:31.088129 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:31.087947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:31.088129 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:31.088114    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:31.088602 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:31.088214    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:35.088189705 +0000 UTC m=+44.602271832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:31.247768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:31.247732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dvljc" event={"ID":"dae8f832-f923-4354-9649-b1ddc523ebf8","Type":"ContainerStarted","Data":"bc5363cef2de7593f47324eb8795f89d05c032bd0800bbc9d1c6c2d39ac5c8d1"}
Sep 09 03:06:31.263145 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:31.263102    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dvljc" podStartSLOduration=1.675630415 podStartE2EDuration="4.263088909s" totalImagesPullingTime="2.587458494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:31.262129293 +0000 UTC m=+40.776211464" watchObservedRunningTime="2026-09-09 03:06:31.263088909 +0000 UTC m=+40.777171059"
Sep 09 03:06:35.114467 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:35.114429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:35.114887 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:35.114544    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:35.114887 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:35.114599    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:43.114585144 +0000 UTC m=+52.628667273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:43.171084 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:43.171033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:43.171625 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:43.171198    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:43.171625 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:43.171271    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:06:59.171251978 +0000 UTC m=+68.685334107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:52.223018 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:52.222991    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-94lfn"
Sep 09 03:06:59.180783 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.180743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:06:59.181295 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:59.180933    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:06:59.181295 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:59.181008    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:07:31.180991601 +0000 UTC m=+100.695073729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:06:59.683798 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.683744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:59.684020 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.683867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:06:59.686415 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.686396    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:06:59.686461 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.686435    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:06:59.694418 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:59.694401    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:06:59.694513 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:06:59.694503    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:08:03.69448877 +0000 UTC m=+133.208570898 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : secret "metrics-daemon-secret" not found
Sep 09 03:06:59.696331 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.696319    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:06:59.708460 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.708440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnwqj\" (UniqueName: \"kubernetes.io/projected/bafde9e9-c0d3-4b04-b0eb-d77c722be9ec-kube-api-access-dnwqj\") pod \"network-check-target-nkqd9\" (UID: \"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec\") " pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:06:59.784932 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.784893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:59.787342 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.787324    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:06:59.797441 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.797416    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e353f605-9075-414d-aa37-b8b53f094c02-original-pull-secret\") pod \"global-pull-secret-syncer-g698z\" (UID: \"e353f605-9075-414d-aa37-b8b53f094c02\") " pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:59.921687 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.921640    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g698z"
Sep 09 03:06:59.929938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.929914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qqmk2\""
Sep 09 03:06:59.937889 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:06:59.937817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:07:00.190465 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:00.190377    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g698z"]
Sep 09 03:07:00.194482 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:07:00.194455    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode353f605_9075_414d_aa37_b8b53f094c02.slice/crio-be3decd775f2b4dfb587f3991a5dcdba7d8948ea703dc0e313d24bf5f0e5cc9f WatchSource:0}: Error finding container be3decd775f2b4dfb587f3991a5dcdba7d8948ea703dc0e313d24bf5f0e5cc9f: Status 404 returned error can't find the container with id be3decd775f2b4dfb587f3991a5dcdba7d8948ea703dc0e313d24bf5f0e5cc9f
Sep 09 03:07:00.211207 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:00.211176    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nkqd9"]
Sep 09 03:07:00.214520 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:07:00.214490    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafde9e9_c0d3_4b04_b0eb_d77c722be9ec.slice/crio-088476315b40e92c8cf3853f17256074122c411bed3b02f0877afdf9efd7acda WatchSource:0}: Error finding container 088476315b40e92c8cf3853f17256074122c411bed3b02f0877afdf9efd7acda: Status 404 returned error can't find the container with id 088476315b40e92c8cf3853f17256074122c411bed3b02f0877afdf9efd7acda
Sep 09 03:07:00.318571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:00.318539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nkqd9" event={"ID":"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec","Type":"ContainerStarted","Data":"088476315b40e92c8cf3853f17256074122c411bed3b02f0877afdf9efd7acda"}
Sep 09 03:07:00.319630 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:00.319600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g698z" event={"ID":"e353f605-9075-414d-aa37-b8b53f094c02","Type":"ContainerStarted","Data":"be3decd775f2b4dfb587f3991a5dcdba7d8948ea703dc0e313d24bf5f0e5cc9f"}
Sep 09 03:07:05.335484 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:05.335442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nkqd9" event={"ID":"bafde9e9-c0d3-4b04-b0eb-d77c722be9ec","Type":"ContainerStarted","Data":"c4fb41e57c0819b7d71893974322a465028b22816081919cdcd1beee57f096f9"}
Sep 09 03:07:05.336002 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:05.335544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:07:05.337041 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:05.337011    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g698z" event={"ID":"e353f605-9075-414d-aa37-b8b53f094c02","Type":"ContainerStarted","Data":"666c5aea44d6ec2b121dc96a6c341dc6ad3ae1b1ee54fe1938b6be0e8d83cecc"}
Sep 09 03:07:05.350273 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:05.350225    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nkqd9" podStartSLOduration=69.962419309 podStartE2EDuration="1m14.350208157s" totalImagesPullingTime="4.387788848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:05.349591589 +0000 UTC m=+74.863673734" watchObservedRunningTime="2026-09-09 03:07:05.350208157 +0000 UTC m=+74.864290307"
Sep 09 03:07:05.365712 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:05.365657    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g698z" podStartSLOduration=69.95182792599999 podStartE2EDuration="1m14.365635723s" totalImagesPullingTime="4.413807797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:05.365539205 +0000 UTC m=+74.879621371" watchObservedRunningTime="2026-09-09 03:07:05.365635723 +0000 UTC m=+74.879717883"
Sep 09 03:07:31.207268 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:31.207231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:07:31.207675 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:07:31.207344    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:07:31.207675 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:07:31.207395    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:08:35.2073817 +0000 UTC m=+164.721463827 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:07:36.341322 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:07:36.341284    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nkqd9"
Sep 09 03:08:03.733253 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:08:03.733217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:08:03.733644 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:08:03.733323    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:08:03.733644 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:08:03.733375    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs podName:734355ce-8feb-4305-8872-543d37a47fc4 nodeName:}" failed. No retries permitted until 2026-09-09 03:10:05.733361403 +0000 UTC m=+255.247443535 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs") pod "network-metrics-daemon-zqzm9" (UID: "734355ce-8feb-4305-8872-543d37a47fc4") : secret "metrics-daemon-secret" not found
Sep 09 03:08:30.325043 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:08:30.324991    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rg75s" podUID="b72822b1-64d7-48a4-9101-4b8dcbb6ef09"
Sep 09 03:08:30.550554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:08:30.550525    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rg75s"
Sep 09 03:08:32.014029 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:08:32.013973    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zqzm9" podUID="734355ce-8feb-4305-8872-543d37a47fc4"
Sep 09 03:08:35.251353 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:08:35.251306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:08:35.251764 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:08:35.251458    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:08:35.251764 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:08:35.251525    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls podName:b72822b1-64d7-48a4-9101-4b8dcbb6ef09 nodeName:}" failed. No retries permitted until 2026-09-09 03:10:37.251508868 +0000 UTC m=+286.765590995 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls") pod "dns-default-rg75s" (UID: "b72822b1-64d7-48a4-9101-4b8dcbb6ef09") : secret "dns-default-metrics-tls" not found
Sep 09 03:08:45.001528 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:08:45.001441    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:09:36.050335 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.050303    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vt567"]
Sep 09 03:09:36.052160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.052144    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dd9bdf694-zvrrf"]
Sep 09 03:09:36.052281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.052264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.054330 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.054307    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.057003 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.056980    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 03:09:36.057109 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.057009    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 03:09:36.057109 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.057040    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.057280 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.057265    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.057349 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.057289    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-phzz9\""
Sep 09 03:09:36.057349 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.057306    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 03:09:36.057349 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.057269    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 03:09:36.057739 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.057722    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nklpr\""
Sep 09 03:09:36.058124 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.058104    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 03:09:36.063407 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.063384    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 03:09:36.064387 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.064373    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 03:09:36.072257 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.072235    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd9bdf694-zvrrf"]
Sep 09 03:09:36.078428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.077016    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vt567"]
Sep 09 03:09:36.137220 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.137191    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8"]
Sep 09 03:09:36.139031 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.139016    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8"
Sep 09 03:09:36.141238 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.141219    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jr5j5\""
Sep 09 03:09:36.141459 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.141442    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.141459 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.141447    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.143956 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.143927    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"]
Sep 09 03:09:36.145870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.145847    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-l8xmz"]
Sep 09 03:09:36.145971 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.145912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.147790 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.147769    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"]
Sep 09 03:09:36.147944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.147892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.148401 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.148381    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vfzdj\""
Sep 09 03:09:36.148733 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.148711    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 03:09:36.148810 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.148772    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.148898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.148882    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.149041 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.149029    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 03:09:36.149608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.149592    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.150466 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.150448    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.150919 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.150893    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.151039 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.150925    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h4gpg\""
Sep 09 03:09:36.151398 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.151346    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 03:09:36.151586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.151568    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8"]
Sep 09 03:09:36.151695 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.151677    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 03:09:36.151967 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.151946    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 03:09:36.151967 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.151960    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4ff8v\""
Sep 09 03:09:36.152235 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.152220    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.152420 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.152399    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.154241 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.154221    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 03:09:36.157505 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.157483    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 03:09:36.162234 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.162210    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-l8xmz"]
Sep 09 03:09:36.162899 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.162876    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"]
Sep 09 03:09:36.166722 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.166700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-trusted-ca\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.166878 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.166732    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-image-registry-private-configuration\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.166878 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.166777    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.166878 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.166803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-serving-cert\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.166901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.166936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88b68\" (UniqueName: \"kubernetes.io/projected/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-kube-api-access-88b68\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.166990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-tmp\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.167019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d487509e-dff4-4996-bacb-7d83f0b599ad-ca-trust-extracted\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.167042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-installation-pull-secrets\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.167075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.167096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-bound-sa-token\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.167120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwv9p\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-kube-api-access-bwv9p\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.167201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-snapshots\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.167284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.167244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-certificates\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.169170 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.169150    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"]
Sep 09 03:09:36.171469 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.171448    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"]
Sep 09 03:09:36.173422 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.173406    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"
Sep 09 03:09:36.175955 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.175906    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 03:09:36.175955 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.175982    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 03:09:36.175955 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.175988    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-t7smg\""
Sep 09 03:09:36.176163 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.176031    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 03:09:36.176163 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.176030    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 03:09:36.189338 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.189314    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"]
Sep 09 03:09:36.267566 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d66383af-0733-4820-b76b-3c4ca43d413b-config\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.267566 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.267767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-serving-cert\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.267767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88b68\" (UniqueName: \"kubernetes.io/projected/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-kube-api-access-88b68\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.267767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d487509e-dff4-4996-bacb-7d83f0b599ad-ca-trust-extracted\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.267767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwxj9\" (UniqueName: \"kubernetes.io/projected/d66383af-0733-4820-b76b-3c4ca43d413b-kube-api-access-mwxj9\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.267767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zbg\" (UniqueName: \"kubernetes.io/projected/019cb9a5-1c1a-474f-b098-85ce52ff122b-kube-api-access-f9zbg\") pod \"volume-data-source-validator-695c895bd6-sgmx8\" (UID: \"019cb9a5-1c1a-474f-b098-85ce52ff122b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8"
Sep 09 03:09:36.268073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.268073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8tq\" (UniqueName: \"kubernetes.io/projected/89139c70-916b-4714-a28e-80202823aa6d-kube-api-access-rw8tq\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.268073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eb6a114-d406-4912-8384-74f5079d9b95-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65467854-qgcqw\" (UID: \"2eb6a114-d406-4912-8384-74f5079d9b95\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"
Sep 09 03:09:36.268073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a215f92-12cb-448a-b375-5a5b668a0998-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.268073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.267994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.268073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-bound-sa-token\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.268073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-snapshots\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-image-registry-private-configuration\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.268096    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.268113    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd9bdf694-zvrrf: secret "image-registry-tls" not found
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268117    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a215f92-12cb-448a-b375-5a5b668a0998-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.268175    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls podName:d487509e-dff4-4996-bacb-7d83f0b599ad nodeName:}" failed. No retries permitted until 2026-09-09 03:09:36.768154457 +0000 UTC m=+226.282236585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls") pod "image-registry-5dd9bdf694-zvrrf" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad") : secret "image-registry-tls" not found
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268194    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d487509e-dff4-4996-bacb-7d83f0b599ad-ca-trust-extracted\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgql\" (UniqueName: \"kubernetes.io/projected/0a215f92-12cb-448a-b375-5a5b668a0998-kube-api-access-wcgql\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-installation-pull-secrets\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-tmp\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.268394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwv9p\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-kube-api-access-bwv9p\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.268915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-trusted-ca\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.268915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d66383af-0733-4820-b76b-3c4ca43d413b-serving-cert\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.268915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-certificates\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.268915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d66383af-0733-4820-b76b-3c4ca43d413b-trusted-ca\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.268915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.268915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqztg\" (UniqueName: \"kubernetes.io/projected/2eb6a114-d406-4912-8384-74f5079d9b95-kube-api-access-gqztg\") pod \"managed-serviceaccount-addon-agent-65467854-qgcqw\" (UID: \"2eb6a114-d406-4912-8384-74f5079d9b95\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"
Sep 09 03:09:36.268915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/89139c70-916b-4714-a28e-80202823aa6d-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.268915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.268910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.269295 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.269095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-certificates\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.269482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.269459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-snapshots\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.269676 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.269654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-tmp\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.269801 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.269784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-trusted-ca\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.270333 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.270312    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-serving-cert\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.270812 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.270791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-installation-pull-secrets\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.271040 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.271020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-image-registry-private-configuration\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.279292 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.279273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwv9p\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-kube-api-access-bwv9p\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.279404 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.279365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-bound-sa-token\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.279404 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.279372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88b68\" (UniqueName: \"kubernetes.io/projected/c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1-kube-api-access-88b68\") pod \"insights-operator-76bcd6ccf5-vt567\" (UID: \"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.362522 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.362439    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vt567"
Sep 09 03:09:36.369359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d66383af-0733-4820-b76b-3c4ca43d413b-serving-cert\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.369424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d66383af-0733-4820-b76b-3c4ca43d413b-trusted-ca\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.369424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqztg\" (UniqueName: \"kubernetes.io/projected/2eb6a114-d406-4912-8384-74f5079d9b95-kube-api-access-gqztg\") pod \"managed-serviceaccount-addon-agent-65467854-qgcqw\" (UID: \"2eb6a114-d406-4912-8384-74f5079d9b95\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"
Sep 09 03:09:36.369424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/89139c70-916b-4714-a28e-80202823aa6d-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.369560 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d66383af-0733-4820-b76b-3c4ca43d413b-config\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.369676 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwxj9\" (UniqueName: \"kubernetes.io/projected/d66383af-0733-4820-b76b-3c4ca43d413b-kube-api-access-mwxj9\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.369789 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zbg\" (UniqueName: \"kubernetes.io/projected/019cb9a5-1c1a-474f-b098-85ce52ff122b-kube-api-access-f9zbg\") pod \"volume-data-source-validator-695c895bd6-sgmx8\" (UID: \"019cb9a5-1c1a-474f-b098-85ce52ff122b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8"
Sep 09 03:09:36.369789 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.369789 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8tq\" (UniqueName: \"kubernetes.io/projected/89139c70-916b-4714-a28e-80202823aa6d-kube-api-access-rw8tq\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.369978 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eb6a114-d406-4912-8384-74f5079d9b95-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65467854-qgcqw\" (UID: \"2eb6a114-d406-4912-8384-74f5079d9b95\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"
Sep 09 03:09:36.369978 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a215f92-12cb-448a-b375-5a5b668a0998-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.369978 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a215f92-12cb-448a-b375-5a5b668a0998-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.369978 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.369946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgql\" (UniqueName: \"kubernetes.io/projected/0a215f92-12cb-448a-b375-5a5b668a0998-kube-api-access-wcgql\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.370518 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.370277    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:36.370518 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.370342    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls podName:89139c70-916b-4714-a28e-80202823aa6d nodeName:}" failed. No retries permitted until 2026-09-09 03:09:36.870322431 +0000 UTC m=+226.384404559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dldlp" (UID: "89139c70-916b-4714-a28e-80202823aa6d") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:36.370518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.370345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/89139c70-916b-4714-a28e-80202823aa6d-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.370518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.370403    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d66383af-0733-4820-b76b-3c4ca43d413b-trusted-ca\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.370891 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.370866    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a215f92-12cb-448a-b375-5a5b668a0998-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.370995 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.370973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d66383af-0733-4820-b76b-3c4ca43d413b-config\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.372266 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.372187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d66383af-0733-4820-b76b-3c4ca43d413b-serving-cert\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.372488 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.372468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a215f92-12cb-448a-b375-5a5b668a0998-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.373122 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.373102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eb6a114-d406-4912-8384-74f5079d9b95-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65467854-qgcqw\" (UID: \"2eb6a114-d406-4912-8384-74f5079d9b95\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"
Sep 09 03:09:36.380335 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.380313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwxj9\" (UniqueName: \"kubernetes.io/projected/d66383af-0733-4820-b76b-3c4ca43d413b-kube-api-access-mwxj9\") pod \"console-operator-85b454874b-l8xmz\" (UID: \"d66383af-0733-4820-b76b-3c4ca43d413b\") " pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.380996 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.380966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zbg\" (UniqueName: \"kubernetes.io/projected/019cb9a5-1c1a-474f-b098-85ce52ff122b-kube-api-access-f9zbg\") pod \"volume-data-source-validator-695c895bd6-sgmx8\" (UID: \"019cb9a5-1c1a-474f-b098-85ce52ff122b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8"
Sep 09 03:09:36.381090 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.381071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8tq\" (UniqueName: \"kubernetes.io/projected/89139c70-916b-4714-a28e-80202823aa6d-kube-api-access-rw8tq\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.381090 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.381078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgql\" (UniqueName: \"kubernetes.io/projected/0a215f92-12cb-448a-b375-5a5b668a0998-kube-api-access-wcgql\") pod \"kube-storage-version-migrator-operator-8d7dd4675-k6749\" (UID: \"0a215f92-12cb-448a-b375-5a5b668a0998\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.381504 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.381486    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqztg\" (UniqueName: \"kubernetes.io/projected/2eb6a114-d406-4912-8384-74f5079d9b95-kube-api-access-gqztg\") pod \"managed-serviceaccount-addon-agent-65467854-qgcqw\" (UID: \"2eb6a114-d406-4912-8384-74f5079d9b95\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"
Sep 09 03:09:36.449343 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.449274    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8"
Sep 09 03:09:36.458038 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.458018    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"
Sep 09 03:09:36.464885 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.464855    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:36.489611 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.489309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"
Sep 09 03:09:36.630213 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.630188    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vt567"]
Sep 09 03:09:36.632657 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:36.632630    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54ade31_ffe0_4cc4_9a3c_b02b5a7523b1.slice/crio-ca5c02763112236b556e071e9be8464b5328d0f5bff77db132484ff56dab1485 WatchSource:0}: Error finding container ca5c02763112236b556e071e9be8464b5328d0f5bff77db132484ff56dab1485: Status 404 returned error can't find the container with id ca5c02763112236b556e071e9be8464b5328d0f5bff77db132484ff56dab1485
Sep 09 03:09:36.709500 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.709468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vt567" event={"ID":"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1","Type":"ContainerStarted","Data":"ca5c02763112236b556e071e9be8464b5328d0f5bff77db132484ff56dab1485"}
Sep 09 03:09:36.725654 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.725632    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8"]
Sep 09 03:09:36.727945 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:36.727914    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019cb9a5_1c1a_474f_b098_85ce52ff122b.slice/crio-3c4b934f23165551092dc4f746c8d80b7afa0d14c24417ad1a8f00af0f4c7ef0 WatchSource:0}: Error finding container 3c4b934f23165551092dc4f746c8d80b7afa0d14c24417ad1a8f00af0f4c7ef0: Status 404 returned error can't find the container with id 3c4b934f23165551092dc4f746c8d80b7afa0d14c24417ad1a8f00af0f4c7ef0
Sep 09 03:09:36.750353 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.750324    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-l8xmz"]
Sep 09 03:09:36.752938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.752914    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749"]
Sep 09 03:09:36.753190 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:36.753166    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66383af_0733_4820_b76b_3c4ca43d413b.slice/crio-1229a62cbab93b0f30f3ef97f64b13ce85f4c0f6c808c4321e2026351b271b4c WatchSource:0}: Error finding container 1229a62cbab93b0f30f3ef97f64b13ce85f4c0f6c808c4321e2026351b271b4c: Status 404 returned error can't find the container with id 1229a62cbab93b0f30f3ef97f64b13ce85f4c0f6c808c4321e2026351b271b4c
Sep 09 03:09:36.755867 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:36.755843    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a215f92_12cb_448a_b375_5a5b668a0998.slice/crio-ea96b5a93f6165ee7ae484fb0e939da2a4c3acac7866898f0f459b687d258d38 WatchSource:0}: Error finding container ea96b5a93f6165ee7ae484fb0e939da2a4c3acac7866898f0f459b687d258d38: Status 404 returned error can't find the container with id ea96b5a93f6165ee7ae484fb0e939da2a4c3acac7866898f0f459b687d258d38
Sep 09 03:09:36.773086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.773065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:36.773222 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.773206    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:09:36.773259 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.773224    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd9bdf694-zvrrf: secret "image-registry-tls" not found
Sep 09 03:09:36.773296 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.773270    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls podName:d487509e-dff4-4996-bacb-7d83f0b599ad nodeName:}" failed. No retries permitted until 2026-09-09 03:09:37.773254532 +0000 UTC m=+227.287336662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls") pod "image-registry-5dd9bdf694-zvrrf" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad") : secret "image-registry-tls" not found
Sep 09 03:09:36.784115 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.784094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw"]
Sep 09 03:09:36.786960 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:36.786936    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eb6a114_d406_4912_8384_74f5079d9b95.slice/crio-1b72172464966a67dc4b47846830acdadc7f7a068c74c08592f9e4367e5b1d3d WatchSource:0}: Error finding container 1b72172464966a67dc4b47846830acdadc7f7a068c74c08592f9e4367e5b1d3d: Status 404 returned error can't find the container with id 1b72172464966a67dc4b47846830acdadc7f7a068c74c08592f9e4367e5b1d3d
Sep 09 03:09:36.874133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:36.874093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:36.874281 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.874252    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:36.874326 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:36.874320    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls podName:89139c70-916b-4714-a28e-80202823aa6d nodeName:}" failed. No retries permitted until 2026-09-09 03:09:37.874304115 +0000 UTC m=+227.388386264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dldlp" (UID: "89139c70-916b-4714-a28e-80202823aa6d") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:37.715311 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:37.715271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8" event={"ID":"019cb9a5-1c1a-474f-b098-85ce52ff122b","Type":"ContainerStarted","Data":"3c4b934f23165551092dc4f746c8d80b7afa0d14c24417ad1a8f00af0f4c7ef0"}
Sep 09 03:09:37.717457 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:37.717400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw" event={"ID":"2eb6a114-d406-4912-8384-74f5079d9b95","Type":"ContainerStarted","Data":"1b72172464966a67dc4b47846830acdadc7f7a068c74c08592f9e4367e5b1d3d"}
Sep 09 03:09:37.719753 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:37.719679    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-l8xmz" event={"ID":"d66383af-0733-4820-b76b-3c4ca43d413b","Type":"ContainerStarted","Data":"1229a62cbab93b0f30f3ef97f64b13ce85f4c0f6c808c4321e2026351b271b4c"}
Sep 09 03:09:37.722052 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:37.721995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749" event={"ID":"0a215f92-12cb-448a-b375-5a5b668a0998","Type":"ContainerStarted","Data":"ea96b5a93f6165ee7ae484fb0e939da2a4c3acac7866898f0f459b687d258d38"}
Sep 09 03:09:37.783805 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:37.783187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:37.783805 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:37.783394    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:09:37.783805 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:37.783412    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd9bdf694-zvrrf: secret "image-registry-tls" not found
Sep 09 03:09:37.783805 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:37.783468    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls podName:d487509e-dff4-4996-bacb-7d83f0b599ad nodeName:}" failed. No retries permitted until 2026-09-09 03:09:39.783448431 +0000 UTC m=+229.297530564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls") pod "image-registry-5dd9bdf694-zvrrf" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad") : secret "image-registry-tls" not found
Sep 09 03:09:37.884651 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:37.884492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:37.884852 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:37.884696    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:37.884852 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:37.884764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls podName:89139c70-916b-4714-a28e-80202823aa6d nodeName:}" failed. No retries permitted until 2026-09-09 03:09:39.88474447 +0000 UTC m=+229.398826597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dldlp" (UID: "89139c70-916b-4714-a28e-80202823aa6d") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:38.726915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:38.726872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8" event={"ID":"019cb9a5-1c1a-474f-b098-85ce52ff122b","Type":"ContainerStarted","Data":"28786f4e4c576b615e009b7881e5e5a60ef4a84510c1ac3615aff2de3b4665b2"}
Sep 09 03:09:38.743083 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:38.742926    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-sgmx8" podStartSLOduration=1.373388796 podStartE2EDuration="2.742901952s" totalImagesPullingTime="1.369513156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:38.741045241 +0000 UTC m=+228.255127415" watchObservedRunningTime="2026-09-09 03:09:38.742901952 +0000 UTC m=+228.256984103"
Sep 09 03:09:39.800415 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:39.800375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:39.800886 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:39.800540    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:09:39.800886 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:39.800554    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd9bdf694-zvrrf: secret "image-registry-tls" not found
Sep 09 03:09:39.800886 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:39.800614    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls podName:d487509e-dff4-4996-bacb-7d83f0b599ad nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.800596028 +0000 UTC m=+233.314678163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls") pod "image-registry-5dd9bdf694-zvrrf" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad") : secret "image-registry-tls" not found
Sep 09 03:09:39.901242 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:39.901193    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:39.901417 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:39.901373    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:39.901471 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:39.901458    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls podName:89139c70-916b-4714-a28e-80202823aa6d nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.90143678 +0000 UTC m=+233.415518912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dldlp" (UID: "89139c70-916b-4714-a28e-80202823aa6d") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:41.370857 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:41.370804    2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66383af_0733_4820_b76b_3c4ca43d413b.slice/crio-5e3b141afea79d9af3e15c2d167edb6fffd2b17cc0d2a9f496f2db75f07a0805.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66383af_0733_4820_b76b_3c4ca43d413b.slice/crio-conmon-5e3b141afea79d9af3e15c2d167edb6fffd2b17cc0d2a9f496f2db75f07a0805.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 03:09:41.738476 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.738435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw" event={"ID":"2eb6a114-d406-4912-8384-74f5079d9b95","Type":"ContainerStarted","Data":"25c449a7b0836f765760048c4b05e86daafe940afd70ce429f9e9c2ce3cd2060"}
Sep 09 03:09:41.740149 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.740122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vt567" event={"ID":"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1","Type":"ContainerStarted","Data":"dd447b4458960d5e9f7676e5d2e7ad4ce555f9a8c4afd95ab70dc876dd8d8f49"}
Sep 09 03:09:41.741925 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.741907    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/0.log"
Sep 09 03:09:41.742033 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.741939    2583 generic.go:356] "Generic (PLEG): container finished" podID="d66383af-0733-4820-b76b-3c4ca43d413b" containerID="5e3b141afea79d9af3e15c2d167edb6fffd2b17cc0d2a9f496f2db75f07a0805" exitCode=255
Sep 09 03:09:41.742033 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.742007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-l8xmz" event={"ID":"d66383af-0733-4820-b76b-3c4ca43d413b","Type":"ContainerDied","Data":"5e3b141afea79d9af3e15c2d167edb6fffd2b17cc0d2a9f496f2db75f07a0805"}
Sep 09 03:09:41.742180 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.742165    2583 scope.go:117] "RemoveContainer" containerID="5e3b141afea79d9af3e15c2d167edb6fffd2b17cc0d2a9f496f2db75f07a0805"
Sep 09 03:09:41.743659 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.743606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749" event={"ID":"0a215f92-12cb-448a-b375-5a5b668a0998","Type":"ContainerStarted","Data":"8ba404be189a7b12de4ef02a0b8f67194ced9ced4907ca81dfc243df866ef331"}
Sep 09 03:09:41.752926 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.752871    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65467854-qgcqw" podStartSLOduration=1.309368806 podStartE2EDuration="5.752853023s" totalImagesPullingTime="4.443484217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:41.751629573 +0000 UTC m=+231.265711725" watchObservedRunningTime="2026-09-09 03:09:41.752853023 +0000 UTC m=+231.266935166"
Sep 09 03:09:41.767615 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.767560    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-vt567" podStartSLOduration=1.238695255 podStartE2EDuration="5.767539838s" totalImagesPullingTime="4.528844583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:41.767036344 +0000 UTC m=+231.281118495" watchObservedRunningTime="2026-09-09 03:09:41.767539838 +0000 UTC m=+231.281621989"
Sep 09 03:09:41.781110 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:41.781062    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749" podStartSLOduration=1.376257852 podStartE2EDuration="5.781041176s" totalImagesPullingTime="4.404783324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:41.780522515 +0000 UTC m=+231.294604665" watchObservedRunningTime="2026-09-09 03:09:41.781041176 +0000 UTC m=+231.295123331"
Sep 09 03:09:42.201907 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.201780    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ncfxg"]
Sep 09 03:09:42.205313 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.205292    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.207665 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.207637    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 03:09:42.207665 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.207648    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 03:09:42.207815 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.207759    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89spd\""
Sep 09 03:09:42.216214 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.216190    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ncfxg"]
Sep 09 03:09:42.322094 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.322047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6c32567-4f89-4461-b0a4-81621726808a-crio-socket\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.322278 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.322114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.322278 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.322221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6c32567-4f89-4461-b0a4-81621726808a-data-volume\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.322278 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.322253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qc9\" (UniqueName: \"kubernetes.io/projected/c6c32567-4f89-4461-b0a4-81621726808a-kube-api-access-95qc9\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.322379 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.322294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.423425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.423378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.423450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6c32567-4f89-4461-b0a4-81621726808a-crio-socket\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.423511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.423535    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.423580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6c32567-4f89-4461-b0a4-81621726808a-crio-socket\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.423585    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.423593    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:42.923577147 +0000 UTC m=+232.437659279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.423690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6c32567-4f89-4461-b0a4-81621726808a-data-volume\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.423713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qc9\" (UniqueName: \"kubernetes.io/projected/c6c32567-4f89-4461-b0a4-81621726808a-kube-api-access-95qc9\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.423814 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.423767    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:42.923740814 +0000 UTC m=+232.437822949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.424200 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.424065    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6c32567-4f89-4461-b0a4-81621726808a-data-volume\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.431754 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.431725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qc9\" (UniqueName: \"kubernetes.io/projected/c6c32567-4f89-4461-b0a4-81621726808a-kube-api-access-95qc9\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.748263 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.748230    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:09:42.748741 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.748721    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/0.log"
Sep 09 03:09:42.748816 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.748768    2583 generic.go:356] "Generic (PLEG): container finished" podID="d66383af-0733-4820-b76b-3c4ca43d413b" containerID="52df2333126ca7561d822d3eced9f9445000ae568e411dd8a7d2d95b85172645" exitCode=255
Sep 09 03:09:42.748890 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.748805    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-l8xmz" event={"ID":"d66383af-0733-4820-b76b-3c4ca43d413b","Type":"ContainerDied","Data":"52df2333126ca7561d822d3eced9f9445000ae568e411dd8a7d2d95b85172645"}
Sep 09 03:09:42.748948 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.748890    2583 scope.go:117] "RemoveContainer" containerID="5e3b141afea79d9af3e15c2d167edb6fffd2b17cc0d2a9f496f2db75f07a0805"
Sep 09 03:09:42.749428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.749411    2583 scope.go:117] "RemoveContainer" containerID="52df2333126ca7561d822d3eced9f9445000ae568e411dd8a7d2d95b85172645"
Sep 09 03:09:42.749576 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.749560    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-l8xmz_openshift-console-operator(d66383af-0733-4820-b76b-3c4ca43d413b)\"" pod="openshift-console-operator/console-operator-85b454874b-l8xmz" podUID="d66383af-0733-4820-b76b-3c4ca43d413b"
Sep 09 03:09:42.775139 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.775116    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv"]
Sep 09 03:09:42.779091 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.779073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv"
Sep 09 03:09:42.781528 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.781506    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 03:09:42.781629 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.781513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kf7hd\""
Sep 09 03:09:42.781629 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.781517    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 03:09:42.786219 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.786199    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv"]
Sep 09 03:09:42.930107 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.930067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.930262 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.930150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzqd\" (UniqueName: \"kubernetes.io/projected/11b8db99-a950-4d63-befb-1ce9d91be8a3-kube-api-access-5bzqd\") pod \"migrator-fdbb8fdff-78cwv\" (UID: \"11b8db99-a950-4d63-befb-1ce9d91be8a3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv"
Sep 09 03:09:42.930262 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:42.930174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:42.930262 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.930211    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.930262 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.930250    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:42.930400 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.930282    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.930264997 +0000 UTC m=+233.444347125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:42.930400 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:42.930298    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:43.930290989 +0000 UTC m=+233.444373117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:43.031285 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.031190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzqd\" (UniqueName: \"kubernetes.io/projected/11b8db99-a950-4d63-befb-1ce9d91be8a3-kube-api-access-5bzqd\") pod \"migrator-fdbb8fdff-78cwv\" (UID: \"11b8db99-a950-4d63-befb-1ce9d91be8a3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv"
Sep 09 03:09:43.039303 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.039278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzqd\" (UniqueName: \"kubernetes.io/projected/11b8db99-a950-4d63-befb-1ce9d91be8a3-kube-api-access-5bzqd\") pod \"migrator-fdbb8fdff-78cwv\" (UID: \"11b8db99-a950-4d63-befb-1ce9d91be8a3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv"
Sep 09 03:09:43.089614 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.089583    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv"
Sep 09 03:09:43.354713 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.354640    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv"]
Sep 09 03:09:43.357578 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:43.357553    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b8db99_a950_4d63_befb_1ce9d91be8a3.slice/crio-8d38f04bf6a82c53fc9038536378afcd12ed130ac438bab577d588db62c6b02a WatchSource:0}: Error finding container 8d38f04bf6a82c53fc9038536378afcd12ed130ac438bab577d588db62c6b02a: Status 404 returned error can't find the container with id 8d38f04bf6a82c53fc9038536378afcd12ed130ac438bab577d588db62c6b02a
Sep 09 03:09:43.753517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.753485    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:09:43.753994 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.753850    2583 scope.go:117] "RemoveContainer" containerID="52df2333126ca7561d822d3eced9f9445000ae568e411dd8a7d2d95b85172645"
Sep 09 03:09:43.754052 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.754006    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-l8xmz_openshift-console-operator(d66383af-0733-4820-b76b-3c4ca43d413b)\"" pod="openshift-console-operator/console-operator-85b454874b-l8xmz" podUID="d66383af-0733-4820-b76b-3c4ca43d413b"
Sep 09 03:09:43.754771 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.754745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv" event={"ID":"11b8db99-a950-4d63-befb-1ce9d91be8a3","Type":"ContainerStarted","Data":"8d38f04bf6a82c53fc9038536378afcd12ed130ac438bab577d588db62c6b02a"}
Sep 09 03:09:43.838134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.838091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:43.838309 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.838220    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:09:43.838309 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.838236    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd9bdf694-zvrrf: secret "image-registry-tls" not found
Sep 09 03:09:43.838309 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.838300    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls podName:d487509e-dff4-4996-bacb-7d83f0b599ad nodeName:}" failed. No retries permitted until 2026-09-09 03:09:51.83827971 +0000 UTC m=+241.352361838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls") pod "image-registry-5dd9bdf694-zvrrf" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad") : secret "image-registry-tls" not found
Sep 09 03:09:43.939428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.939388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:43.939601 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.939447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:43.939601 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.939467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:43.939601 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.939545    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:43.939601 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.939543    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:43.939601 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.939583    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:43.939601 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.939594    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:45.939580936 +0000 UTC m=+235.453663065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:43.939601 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.939607    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:45.939601043 +0000 UTC m=+235.453683171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:43.939881 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:43.939625    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls podName:89139c70-916b-4714-a28e-80202823aa6d nodeName:}" failed. No retries permitted until 2026-09-09 03:09:51.93961223 +0000 UTC m=+241.453694363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dldlp" (UID: "89139c70-916b-4714-a28e-80202823aa6d") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:09:43.975402 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:43.975379    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9zhgm_742fbb48-0057-4d26-9b3c-2493b94a952e/dns-node-resolver/0.log"
Sep 09 03:09:44.973724 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:44.973648    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2p8lk_9b96da89-37da-4ea3-a3f6-33282f05e059/node-ca/0.log"
Sep 09 03:09:45.763292 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:45.763254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv" event={"ID":"11b8db99-a950-4d63-befb-1ce9d91be8a3","Type":"ContainerStarted","Data":"5e135e7254f806d35126c7ce03222a2a10b98cece97f34344290942a3a3793a9"}
Sep 09 03:09:45.763439 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:45.763312    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv" event={"ID":"11b8db99-a950-4d63-befb-1ce9d91be8a3","Type":"ContainerStarted","Data":"38958951843b328032394aca0b2af1789c82239db6e7afd8595c3b229a6e6c6e"}
Sep 09 03:09:45.779615 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:45.779564    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-78cwv" podStartSLOduration=2.413903423 podStartE2EDuration="3.779549002s" totalImagesPullingTime="1.365645579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:45.777815607 +0000 UTC m=+235.291897784" watchObservedRunningTime="2026-09-09 03:09:45.779549002 +0000 UTC m=+235.293631182"
Sep 09 03:09:45.956762 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:45.956734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:45.956923 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:45.956807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:45.956923 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:45.956869    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:45.957008 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:45.956925    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:49.956910682 +0000 UTC m=+239.470992809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:45.957008 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:45.956937    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:45.957008 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:45.956977    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:49.956962706 +0000 UTC m=+239.471044835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:45.974656 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:45.974629    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dvljc_dae8f832-f923-4354-9649-b1ddc523ebf8/serve-healthcheck-canary/0.log"
Sep 09 03:09:46.465675 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:46.465629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:46.465881 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:46.465691    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:09:46.466060 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:46.466047    2583 scope.go:117] "RemoveContainer" containerID="52df2333126ca7561d822d3eced9f9445000ae568e411dd8a7d2d95b85172645"
Sep 09 03:09:46.466208 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:46.466191    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-l8xmz_openshift-console-operator(d66383af-0733-4820-b76b-3c4ca43d413b)\"" pod="openshift-console-operator/console-operator-85b454874b-l8xmz" podUID="d66383af-0733-4820-b76b-3c4ca43d413b"
Sep 09 03:09:49.989158 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:49.989109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:49.989552 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:49.989204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:49.989552 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:49.989246    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:09:49.989552 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:49.989299    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:49.989552 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:49.989325    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:57.989306031 +0000 UTC m=+247.503388179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : configmap "kube-rbac-proxy" not found
Sep 09 03:09:49.989552 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:09:49.989343    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls podName:c6c32567-4f89-4461-b0a4-81621726808a nodeName:}" failed. No retries permitted until 2026-09-09 03:09:57.989332419 +0000 UTC m=+247.503414546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncfxg" (UID: "c6c32567-4f89-4461-b0a4-81621726808a") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:09:51.904903 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:51.904859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:51.907475 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:51.907451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"image-registry-5dd9bdf694-zvrrf\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") " pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:51.970666 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:51.970633    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nklpr\""
Sep 09 03:09:51.978814 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:51.978788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:52.005537 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.005498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:52.007944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.007914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89139c70-916b-4714-a28e-80202823aa6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dldlp\" (UID: \"89139c70-916b-4714-a28e-80202823aa6d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:52.074647 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.074613    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4ff8v\""
Sep 09 03:09:52.082623 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.082604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"
Sep 09 03:09:52.251509 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.251340    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd9bdf694-zvrrf"]
Sep 09 03:09:52.253858 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:52.253811    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd487509e_dff4_4996_bacb_7d83f0b599ad.slice/crio-3d614bdf22106ae91ed3e3bfcb07e6dfa7632b4b74866de8b287d1e848bc6d72 WatchSource:0}: Error finding container 3d614bdf22106ae91ed3e3bfcb07e6dfa7632b4b74866de8b287d1e848bc6d72: Status 404 returned error can't find the container with id 3d614bdf22106ae91ed3e3bfcb07e6dfa7632b4b74866de8b287d1e848bc6d72
Sep 09 03:09:52.350923 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.350894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp"]
Sep 09 03:09:52.353867 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:52.353818    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89139c70_916b_4714_a28e_80202823aa6d.slice/crio-5eae19b50bce56958d5700ce1d6e5eecd45b2c368b415fbb18dd6c697c1f7877 WatchSource:0}: Error finding container 5eae19b50bce56958d5700ce1d6e5eecd45b2c368b415fbb18dd6c697c1f7877: Status 404 returned error can't find the container with id 5eae19b50bce56958d5700ce1d6e5eecd45b2c368b415fbb18dd6c697c1f7877
Sep 09 03:09:52.788880 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.788823    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp" event={"ID":"89139c70-916b-4714-a28e-80202823aa6d","Type":"ContainerStarted","Data":"5eae19b50bce56958d5700ce1d6e5eecd45b2c368b415fbb18dd6c697c1f7877"}
Sep 09 03:09:52.790406 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.790380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf" event={"ID":"d487509e-dff4-4996-bacb-7d83f0b599ad","Type":"ContainerStarted","Data":"a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7"}
Sep 09 03:09:52.790552 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.790421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf" event={"ID":"d487509e-dff4-4996-bacb-7d83f0b599ad","Type":"ContainerStarted","Data":"3d614bdf22106ae91ed3e3bfcb07e6dfa7632b4b74866de8b287d1e848bc6d72"}
Sep 09 03:09:52.790552 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.790508    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:09:52.809842 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:52.809779    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf" podStartSLOduration=16.809764006 podStartE2EDuration="16.809764006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:52.809005625 +0000 UTC m=+242.323087795" watchObservedRunningTime="2026-09-09 03:09:52.809764006 +0000 UTC m=+242.323846190"
Sep 09 03:09:54.800301 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:54.800269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp" event={"ID":"89139c70-916b-4714-a28e-80202823aa6d","Type":"ContainerStarted","Data":"7db27995c7af724df99a5e5008e546c182bf2559d6cc2c02766a270c8d48dcb4"}
Sep 09 03:09:54.815291 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:54.815243    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dldlp" podStartSLOduration=16.981394973 podStartE2EDuration="18.815227975s" totalImagesPullingTime="1.833833002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:09:54.815148092 +0000 UTC m=+244.329230267" watchObservedRunningTime="2026-09-09 03:09:54.815227975 +0000 UTC m=+244.329310125"
Sep 09 03:09:58.056575 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.056521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:58.057116 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.056597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:58.057171 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.057150    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6c32567-4f89-4461-b0a4-81621726808a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:58.059219 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.059194    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6c32567-4f89-4461-b0a4-81621726808a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncfxg\" (UID: \"c6c32567-4f89-4461-b0a4-81621726808a\") " pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:58.119074 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.119034    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89spd\""
Sep 09 03:09:58.127122 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.127099    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ncfxg"
Sep 09 03:09:58.400061 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.399891    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ncfxg"]
Sep 09 03:09:58.402884 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:09:58.402853    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c32567_4f89_4461_b0a4_81621726808a.slice/crio-543b542517c8b443a5753867a30984bac138565dfab44c2b774cdca3e7257571 WatchSource:0}: Error finding container 543b542517c8b443a5753867a30984bac138565dfab44c2b774cdca3e7257571: Status 404 returned error can't find the container with id 543b542517c8b443a5753867a30984bac138565dfab44c2b774cdca3e7257571
Sep 09 03:09:58.815948 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.815858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncfxg" event={"ID":"c6c32567-4f89-4461-b0a4-81621726808a","Type":"ContainerStarted","Data":"db34b18264d45b433eb6c085b9e094fef48f058af6253438686ace1cf01765f2"}
Sep 09 03:09:58.815948 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:58.815912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncfxg" event={"ID":"c6c32567-4f89-4461-b0a4-81621726808a","Type":"ContainerStarted","Data":"543b542517c8b443a5753867a30984bac138565dfab44c2b774cdca3e7257571"}
Sep 09 03:09:59.821238 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:09:59.821203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncfxg" event={"ID":"c6c32567-4f89-4461-b0a4-81621726808a","Type":"ContainerStarted","Data":"3ffcd88bcb79e85e038a5cd1c5a34a597c3a6aa65477cf8289d692ff9dbff3b6"}
Sep 09 03:10:00.002228 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:00.002201    2583 scope.go:117] "RemoveContainer" containerID="52df2333126ca7561d822d3eced9f9445000ae568e411dd8a7d2d95b85172645"
Sep 09 03:10:00.827075 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:00.827047    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:10:00.827549 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:00.827163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-l8xmz" event={"ID":"d66383af-0733-4820-b76b-3c4ca43d413b","Type":"ContainerStarted","Data":"d7519c9396c8a14b6d902d694ec6a7b4a029aa5e55615a6498fe6172101855ca"}
Sep 09 03:10:00.827549 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:00.827482    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:10:00.829528 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:00.829500    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncfxg" event={"ID":"c6c32567-4f89-4461-b0a4-81621726808a","Type":"ContainerStarted","Data":"b771dd1ed5fe7ba3a1b7b143cb5826c10e46e362c845d63ae98d7d0885693bf9"}
Sep 09 03:10:00.842938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:00.842887    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-l8xmz" podStartSLOduration=20.435002004 podStartE2EDuration="24.842870687s" totalImagesPullingTime="4.407868683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:00.841917746 +0000 UTC m=+250.355999907" watchObservedRunningTime="2026-09-09 03:10:00.842870687 +0000 UTC m=+250.356952837"
Sep 09 03:10:00.856696 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:00.856632    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ncfxg" podStartSLOduration=16.98463812 podStartE2EDuration="18.856616223s" totalImagesPullingTime="1.871978103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:09:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:00.856371609 +0000 UTC m=+250.370453769" watchObservedRunningTime="2026-09-09 03:10:00.856616223 +0000 UTC m=+250.370698351"
Sep 09 03:10:01.440821 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:01.440792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-l8xmz"
Sep 09 03:10:02.433007 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.432969    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-dp6tq"]
Sep 09 03:10:02.436133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.436113    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-dp6tq"
Sep 09 03:10:02.440664 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.440642    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 03:10:02.440914 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.440891    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5bjsg\""
Sep 09 03:10:02.441073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.440952    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 03:10:02.441970 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.441951    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"]
Sep 09 03:10:02.444974 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.444957    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"
Sep 09 03:10:02.447448 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.447433    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 03:10:02.447554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.447535    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rbzcw\""
Sep 09 03:10:02.453501 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.453480    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dd9bdf694-zvrrf"]
Sep 09 03:10:02.454923 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.454897    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-dp6tq"]
Sep 09 03:10:02.458354 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.458332    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"]
Sep 09 03:10:02.495536 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.495504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2b4\" (UniqueName: \"kubernetes.io/projected/a5fce05f-0b08-457d-935a-b6b30a5c83fc-kube-api-access-gr2b4\") pod \"downloads-85d98c998f-dp6tq\" (UID: \"a5fce05f-0b08-457d-935a-b6b30a5c83fc\") " pod="openshift-console/downloads-85d98c998f-dp6tq"
Sep 09 03:10:02.497385 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.497355    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54bf464b6f-7wmhs"]
Sep 09 03:10:02.500567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.500552    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.535418 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.535385    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54bf464b6f-7wmhs"]
Sep 09 03:10:02.596081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-registry-tls\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.596260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f725e7d0-9c83-4232-92d7-dab1bf647b30-ca-trust-extracted\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.596260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2b4\" (UniqueName: \"kubernetes.io/projected/a5fce05f-0b08-457d-935a-b6b30a5c83fc-kube-api-access-gr2b4\") pod \"downloads-85d98c998f-dp6tq\" (UID: \"a5fce05f-0b08-457d-935a-b6b30a5c83fc\") " pod="openshift-console/downloads-85d98c998f-dp6tq"
Sep 09 03:10:02.596260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f725e7d0-9c83-4232-92d7-dab1bf647b30-trusted-ca\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.596260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f725e7d0-9c83-4232-92d7-dab1bf647b30-installation-pull-secrets\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.596260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f725e7d0-9c83-4232-92d7-dab1bf647b30-registry-certificates\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.596260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-bound-sa-token\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.596517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzc6\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-kube-api-access-tvzc6\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.596517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a88b5550-e0ea-4fdc-8e53-706f27af9828-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-z9vgj\" (UID: \"a88b5550-e0ea-4fdc-8e53-706f27af9828\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"
Sep 09 03:10:02.596517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.596414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f725e7d0-9c83-4232-92d7-dab1bf647b30-image-registry-private-configuration\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.608173 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.608146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2b4\" (UniqueName: \"kubernetes.io/projected/a5fce05f-0b08-457d-935a-b6b30a5c83fc-kube-api-access-gr2b4\") pod \"downloads-85d98c998f-dp6tq\" (UID: \"a5fce05f-0b08-457d-935a-b6b30a5c83fc\") " pod="openshift-console/downloads-85d98c998f-dp6tq"
Sep 09 03:10:02.697125 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-registry-tls\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.697125 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f725e7d0-9c83-4232-92d7-dab1bf647b30-ca-trust-extracted\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.697362 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f725e7d0-9c83-4232-92d7-dab1bf647b30-trusted-ca\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.697362 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f725e7d0-9c83-4232-92d7-dab1bf647b30-installation-pull-secrets\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.697362 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f725e7d0-9c83-4232-92d7-dab1bf647b30-registry-certificates\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.697362 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-bound-sa-token\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.697362 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzc6\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-kube-api-access-tvzc6\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.697362 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a88b5550-e0ea-4fdc-8e53-706f27af9828-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-z9vgj\" (UID: \"a88b5550-e0ea-4fdc-8e53-706f27af9828\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"
Sep 09 03:10:02.697362 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f725e7d0-9c83-4232-92d7-dab1bf647b30-image-registry-private-configuration\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.697683 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.697536    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f725e7d0-9c83-4232-92d7-dab1bf647b30-ca-trust-extracted\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.698233 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.698164    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f725e7d0-9c83-4232-92d7-dab1bf647b30-registry-certificates\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.698583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.698554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f725e7d0-9c83-4232-92d7-dab1bf647b30-trusted-ca\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.699914 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.699893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f725e7d0-9c83-4232-92d7-dab1bf647b30-installation-pull-secrets\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.699994 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.699937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f725e7d0-9c83-4232-92d7-dab1bf647b30-image-registry-private-configuration\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.700127 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.700113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-registry-tls\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.700184 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.700167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a88b5550-e0ea-4fdc-8e53-706f27af9828-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-z9vgj\" (UID: \"a88b5550-e0ea-4fdc-8e53-706f27af9828\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"
Sep 09 03:10:02.705712 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.705686    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-bound-sa-token\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.706179 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.706153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzc6\" (UniqueName: \"kubernetes.io/projected/f725e7d0-9c83-4232-92d7-dab1bf647b30-kube-api-access-tvzc6\") pod \"image-registry-54bf464b6f-7wmhs\" (UID: \"f725e7d0-9c83-4232-92d7-dab1bf647b30\") " pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:02.746136 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.746115    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-dp6tq"
Sep 09 03:10:02.754439 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.754417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"
Sep 09 03:10:02.809428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:02.809383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:03.019569 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.019498    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-dp6tq"]
Sep 09 03:10:03.022265 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:10:03.022235    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fce05f_0b08_457d_935a_b6b30a5c83fc.slice/crio-199b55f046390517a0fb5273d78354180b95302448a7e4c8d776e511ae38c302 WatchSource:0}: Error finding container 199b55f046390517a0fb5273d78354180b95302448a7e4c8d776e511ae38c302: Status 404 returned error can't find the container with id 199b55f046390517a0fb5273d78354180b95302448a7e4c8d776e511ae38c302
Sep 09 03:10:03.025791 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.025705    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"]
Sep 09 03:10:03.028025 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:10:03.028003    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88b5550_e0ea_4fdc_8e53_706f27af9828.slice/crio-0f3609bf4a65b85c8b120fa7c27ba86ac4219ec88e6ed335d5d2621ecd80c794 WatchSource:0}: Error finding container 0f3609bf4a65b85c8b120fa7c27ba86ac4219ec88e6ed335d5d2621ecd80c794: Status 404 returned error can't find the container with id 0f3609bf4a65b85c8b120fa7c27ba86ac4219ec88e6ed335d5d2621ecd80c794
Sep 09 03:10:03.086290 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.086239    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54bf464b6f-7wmhs"]
Sep 09 03:10:03.088547 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:10:03.088528    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf725e7d0_9c83_4232_92d7_dab1bf647b30.slice/crio-f858977b79306b1ca70a87f6b778389d5651c57adf53b9ee4b4ffa3cdff77950 WatchSource:0}: Error finding container f858977b79306b1ca70a87f6b778389d5651c57adf53b9ee4b4ffa3cdff77950: Status 404 returned error can't find the container with id f858977b79306b1ca70a87f6b778389d5651c57adf53b9ee4b4ffa3cdff77950
Sep 09 03:10:03.842348 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.842305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" event={"ID":"f725e7d0-9c83-4232-92d7-dab1bf647b30","Type":"ContainerStarted","Data":"88a66b4d651fb842eb3037c592acece2b1239b9154cc4416fba87d7506d272c3"}
Sep 09 03:10:03.842814 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.842371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" event={"ID":"f725e7d0-9c83-4232-92d7-dab1bf647b30","Type":"ContainerStarted","Data":"f858977b79306b1ca70a87f6b778389d5651c57adf53b9ee4b4ffa3cdff77950"}
Sep 09 03:10:03.842814 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.842449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:03.844354 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.844326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj" event={"ID":"a88b5550-e0ea-4fdc-8e53-706f27af9828","Type":"ContainerStarted","Data":"0f3609bf4a65b85c8b120fa7c27ba86ac4219ec88e6ed335d5d2621ecd80c794"}
Sep 09 03:10:03.845778 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.845751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-dp6tq" event={"ID":"a5fce05f-0b08-457d-935a-b6b30a5c83fc","Type":"ContainerStarted","Data":"199b55f046390517a0fb5273d78354180b95302448a7e4c8d776e511ae38c302"}
Sep 09 03:10:03.861776 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:03.861733    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podStartSLOduration=1.861720514 podStartE2EDuration="1.861720514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:03.860779082 +0000 UTC m=+253.374861236" watchObservedRunningTime="2026-09-09 03:10:03.861720514 +0000 UTC m=+253.375802654"
Sep 09 03:10:04.851461 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:04.851414    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj" event={"ID":"a88b5550-e0ea-4fdc-8e53-706f27af9828","Type":"ContainerStarted","Data":"d0d0d1c98f0717efe2f663a987d6b5f5b2ed648b2a98031437ba40be7fa266b3"}
Sep 09 03:10:04.866607 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:04.866537    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj" podStartSLOduration=1.631349685 podStartE2EDuration="2.866518576s" totalImagesPullingTime="1.235168891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:04.865515622 +0000 UTC m=+254.379597795" watchObservedRunningTime="2026-09-09 03:10:04.866518576 +0000 UTC m=+254.380600726"
Sep 09 03:10:05.823489 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:05.823450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:10:05.826344 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:05.826323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734355ce-8feb-4305-8872-543d37a47fc4-metrics-certs\") pod \"network-metrics-daemon-zqzm9\" (UID: \"734355ce-8feb-4305-8872-543d37a47fc4\") " pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:10:05.855971 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:05.855901    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"
Sep 09 03:10:05.861477 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:05.861452    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-z9vgj"
Sep 09 03:10:06.004310 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:06.004280    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bzb6q\""
Sep 09 03:10:06.012290 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:06.012267    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqzm9"
Sep 09 03:10:06.298561 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:06.294124    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zqzm9"]
Sep 09 03:10:06.861841 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:06.861798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqzm9" event={"ID":"734355ce-8feb-4305-8872-543d37a47fc4","Type":"ContainerStarted","Data":"efc68276b80c80380f2977dd3cdab56a16289d38efd0cd53b0e5bbc54f4ac1f4"}
Sep 09 03:10:07.867818 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:07.867780    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqzm9" event={"ID":"734355ce-8feb-4305-8872-543d37a47fc4","Type":"ContainerStarted","Data":"28f86430f914763289a6133885d2ab621161e01280749321499c3c2bef3c6dde"}
Sep 09 03:10:07.868317 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:07.867858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqzm9" event={"ID":"734355ce-8feb-4305-8872-543d37a47fc4","Type":"ContainerStarted","Data":"3808eaf76242746629e8bb8c62bf2a9aa62422060267ae00698ef6304b234036"}
Sep 09 03:10:07.886137 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:07.886075    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zqzm9" podStartSLOduration=255.965694492 podStartE2EDuration="4m16.886054709s" totalImagesPullingTime="920.360217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:07.88529455 +0000 UTC m=+257.399376702" watchObservedRunningTime="2026-09-09 03:10:07.886054709 +0000 UTC m=+257.400136889"
Sep 09 03:10:11.123594 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.123561    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-746hs"]
Sep 09 03:10:11.126727 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.126711    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.129295 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.129257    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 03:10:11.129436 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.129409    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 03:10:11.129533 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.129513    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 03:10:11.129614 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.129519    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 03:10:11.130293 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.130268    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tzldt\""
Sep 09 03:10:11.170338 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-textfile\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.170338 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-wtmp\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.170553 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchpm\" (UniqueName: \"kubernetes.io/projected/db96c9f0-a549-47ee-ba38-fa42f5296d55-kube-api-access-zchpm\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.170553 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170501    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-accelerators-collector-config\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.170637 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-tls\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.170637 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db96c9f0-a549-47ee-ba38-fa42f5296d55-metrics-client-ca\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.170714 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-sys\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.170714 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-root\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.170714 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.170684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.271621 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-tls\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.271621 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db96c9f0-a549-47ee-ba38-fa42f5296d55-metrics-client-ca\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.271879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-sys\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.271879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-root\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.271879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-root\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.271879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.271879 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:10:11.271808    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 03:10:11.271879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-textfile\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.272187 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:10:11.271889    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-tls podName:db96c9f0-a549-47ee-ba38-fa42f5296d55 nodeName:}" failed. No retries permitted until 2026-09-09 03:10:11.77186785 +0000 UTC m=+261.285949982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-tls") pod "node-exporter-746hs" (UID: "db96c9f0-a549-47ee-ba38-fa42f5296d55") : secret "node-exporter-tls" not found
Sep 09 03:10:11.272187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-sys\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.272187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.271957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-wtmp\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.272187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.272032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchpm\" (UniqueName: \"kubernetes.io/projected/db96c9f0-a549-47ee-ba38-fa42f5296d55-kube-api-access-zchpm\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.272187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.272080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-accelerators-collector-config\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.272187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.272095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-wtmp\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.272589 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.272567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-textfile\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.272745 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.272718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db96c9f0-a549-47ee-ba38-fa42f5296d55-metrics-client-ca\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.272882 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.272778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-accelerators-collector-config\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.274613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.274590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.280984 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.280687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchpm\" (UniqueName: \"kubernetes.io/projected/db96c9f0-a549-47ee-ba38-fa42f5296d55-kube-api-access-zchpm\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.776357 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.776319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-tls\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:11.779190 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:11.779162    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db96c9f0-a549-47ee-ba38-fa42f5296d55-node-exporter-tls\") pod \"node-exporter-746hs\" (UID: \"db96c9f0-a549-47ee-ba38-fa42f5296d55\") " pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:12.039182 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:12.039086    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-746hs"
Sep 09 03:10:12.053256 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:10:12.053221    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb96c9f0_a549_47ee_ba38_fa42f5296d55.slice/crio-652f908bf365f424745ee740a0feee05978d8034457f31becf63e9ec3890c6b5 WatchSource:0}: Error finding container 652f908bf365f424745ee740a0feee05978d8034457f31becf63e9ec3890c6b5: Status 404 returned error can't find the container with id 652f908bf365f424745ee740a0feee05978d8034457f31becf63e9ec3890c6b5
Sep 09 03:10:12.459458 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:12.459414    2583 patch_prober.go:28] interesting pod/image-registry-5dd9bdf694-zvrrf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:12.459940 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:12.459486    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf" podUID="d487509e-dff4-4996-bacb-7d83f0b599ad" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:12.888352 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:12.888301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-746hs" event={"ID":"db96c9f0-a549-47ee-ba38-fa42f5296d55","Type":"ContainerStarted","Data":"652f908bf365f424745ee740a0feee05978d8034457f31becf63e9ec3890c6b5"}
Sep 09 03:10:15.859747 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:15.859715    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"]
Sep 09 03:10:15.889772 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:15.889727    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"]
Sep 09 03:10:15.889947 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:15.889873    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:15.892768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:15.892587    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 03:10:15.892768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:15.892645    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-szb5l\""
Sep 09 03:10:16.014725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.014687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28cs\" (UniqueName: \"kubernetes.io/projected/9c67e593-d7dc-401b-8e6a-61a92a35100e-kube-api-access-l28cs\") pod \"monitoring-plugin-85bbbbbf4-gn28j\" (UID: \"9c67e593-d7dc-401b-8e6a-61a92a35100e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:16.014937 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.014749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c67e593-d7dc-401b-8e6a-61a92a35100e-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-gn28j\" (UID: \"9c67e593-d7dc-401b-8e6a-61a92a35100e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:16.115972 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.115930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28cs\" (UniqueName: \"kubernetes.io/projected/9c67e593-d7dc-401b-8e6a-61a92a35100e-kube-api-access-l28cs\") pod \"monitoring-plugin-85bbbbbf4-gn28j\" (UID: \"9c67e593-d7dc-401b-8e6a-61a92a35100e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:16.116136 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.115995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c67e593-d7dc-401b-8e6a-61a92a35100e-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-gn28j\" (UID: \"9c67e593-d7dc-401b-8e6a-61a92a35100e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:16.116207 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:10:16.116177    2583 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 03:10:16.116274 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:10:16.116260    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c67e593-d7dc-401b-8e6a-61a92a35100e-monitoring-plugin-cert podName:9c67e593-d7dc-401b-8e6a-61a92a35100e nodeName:}" failed. No retries permitted until 2026-09-09 03:10:16.616237052 +0000 UTC m=+266.130319201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9c67e593-d7dc-401b-8e6a-61a92a35100e-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-gn28j" (UID: "9c67e593-d7dc-401b-8e6a-61a92a35100e") : secret "monitoring-plugin-cert" not found
Sep 09 03:10:16.129806 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.129771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28cs\" (UniqueName: \"kubernetes.io/projected/9c67e593-d7dc-401b-8e6a-61a92a35100e-kube-api-access-l28cs\") pod \"monitoring-plugin-85bbbbbf4-gn28j\" (UID: \"9c67e593-d7dc-401b-8e6a-61a92a35100e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:16.290648 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.290615    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d8c95897f-7rfjg"]
Sep 09 03:10:16.307413 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.307353    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d8c95897f-7rfjg"]
Sep 09 03:10:16.307577 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.307499    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.310212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.309932    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 03:10:16.310212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.309991    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 03:10:16.310212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.310101    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 03:10:16.310461 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.310297    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 03:10:16.310938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.310745    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mmdds\""
Sep 09 03:10:16.310938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.310751    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 03:10:16.315781 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.315758    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 03:10:16.419345 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.419255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-metrics-client-ca\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.419345 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.419311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-federate-client-tls\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.419672 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.419640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.419807 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.419691    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncdxz\" (UniqueName: \"kubernetes.io/projected/735ee480-c750-4fc9-baa2-93e5a6ae41ff-kube-api-access-ncdxz\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.419807 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.419772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-telemeter-client-tls\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.419932 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.419815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-serving-certs-ca-bundle\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.419932 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.419857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-secret-telemeter-client\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.419932 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.419908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.521030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.520991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-serving-certs-ca-bundle\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.521030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.521035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-secret-telemeter-client\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.521286 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.521057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.521286 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.521077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-metrics-client-ca\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.521286 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.521117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-federate-client-tls\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.521286 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.521170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.521286 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.521206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncdxz\" (UniqueName: \"kubernetes.io/projected/735ee480-c750-4fc9-baa2-93e5a6ae41ff-kube-api-access-ncdxz\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.521286 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.521261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-telemeter-client-tls\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.522285 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.521761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-serving-certs-ca-bundle\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.522285 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.522250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-metrics-client-ca\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.523176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.523145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735ee480-c750-4fc9-baa2-93e5a6ae41ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.524271 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.524251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-telemeter-client-tls\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.524359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.524268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-secret-telemeter-client\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.524359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.524326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.524799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.524779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/735ee480-c750-4fc9-baa2-93e5a6ae41ff-federate-client-tls\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.529365 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.529342    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncdxz\" (UniqueName: \"kubernetes.io/projected/735ee480-c750-4fc9-baa2-93e5a6ae41ff-kube-api-access-ncdxz\") pod \"telemeter-client-d8c95897f-7rfjg\" (UID: \"735ee480-c750-4fc9-baa2-93e5a6ae41ff\") " pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.621439 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.621398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg"
Sep 09 03:10:16.622689 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.622658    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c67e593-d7dc-401b-8e6a-61a92a35100e-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-gn28j\" (UID: \"9c67e593-d7dc-401b-8e6a-61a92a35100e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:16.625589 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.625559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c67e593-d7dc-401b-8e6a-61a92a35100e-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-gn28j\" (UID: \"9c67e593-d7dc-401b-8e6a-61a92a35100e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:16.801800 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:16.801720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:19.720570 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.720538    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-688d9c44c-hfhb4"]
Sep 09 03:10:19.737119 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.737089    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-688d9c44c-hfhb4"]
Sep 09 03:10:19.737268 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.737199    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.739804 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.739767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 03:10:19.739979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.739771    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 03:10:19.739979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.739775    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 03:10:19.740102 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.739990    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 03:10:19.740735 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.740715    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 03:10:19.740903 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.740884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4b5tb\""
Sep 09 03:10:19.746254 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.746234    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 03:10:19.850082 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.850051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-trusted-ca-bundle\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.850277 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.850098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-oauth-config\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.850277 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.850124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-serving-cert\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.850277 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.850225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-config\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.850277 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.850255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cx8z\" (UniqueName: \"kubernetes.io/projected/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-kube-api-access-7cx8z\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.850496 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.850297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-service-ca\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.850496 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.850353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-oauth-serving-cert\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.951760 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.951728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-config\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.951966 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.951769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cx8z\" (UniqueName: \"kubernetes.io/projected/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-kube-api-access-7cx8z\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.951966 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.951811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-service-ca\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.951966 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.951869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-oauth-serving-cert\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.951966 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.951905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-trusted-ca-bundle\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.951966 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.951950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-oauth-config\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.952221 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.951978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-serving-cert\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.952561 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.952530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-config\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.952853 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.952575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-service-ca\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.952853 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.952579    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-oauth-serving-cert\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.952853 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.952754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-trusted-ca-bundle\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.954475 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.954443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-oauth-config\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.954633 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.954615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-serving-cert\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:19.959531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:19.959509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cx8z\" (UniqueName: \"kubernetes.io/projected/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-kube-api-access-7cx8z\") pod \"console-688d9c44c-hfhb4\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") " pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:20.049452 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.049371    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:20.582564 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.582500    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"]
Sep 09 03:10:20.598095 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.598066    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-688d9c44c-hfhb4"]
Sep 09 03:10:20.602134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.602114    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d8c95897f-7rfjg"]
Sep 09 03:10:20.614976 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:10:20.614952    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c67e593_d7dc_401b_8e6a_61a92a35100e.slice/crio-79c61619c18328ac283730d9fe699d9064578bf545031353876bbc5ff964e223 WatchSource:0}: Error finding container 79c61619c18328ac283730d9fe699d9064578bf545031353876bbc5ff964e223: Status 404 returned error can't find the container with id 79c61619c18328ac283730d9fe699d9064578bf545031353876bbc5ff964e223
Sep 09 03:10:20.647018 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:10:20.646951    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a0c67c_7b98_4d77_9e15_04ddc9f36cdd.slice/crio-0659f68e9dfe05763f831581cecdfa1715014e51cf86764a19acd14a8e6b1aaf WatchSource:0}: Error finding container 0659f68e9dfe05763f831581cecdfa1715014e51cf86764a19acd14a8e6b1aaf: Status 404 returned error can't find the container with id 0659f68e9dfe05763f831581cecdfa1715014e51cf86764a19acd14a8e6b1aaf
Sep 09 03:10:20.647201 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:10:20.647180    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735ee480_c750_4fc9_baa2_93e5a6ae41ff.slice/crio-a3e27b7d6908e5e7088e158ef6c0f343e1589cbcf850efe6e8611e2d188d4679 WatchSource:0}: Error finding container a3e27b7d6908e5e7088e158ef6c0f343e1589cbcf850efe6e8611e2d188d4679: Status 404 returned error can't find the container with id a3e27b7d6908e5e7088e158ef6c0f343e1589cbcf850efe6e8611e2d188d4679
Sep 09 03:10:20.927040 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.926996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688d9c44c-hfhb4" event={"ID":"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd","Type":"ContainerStarted","Data":"0659f68e9dfe05763f831581cecdfa1715014e51cf86764a19acd14a8e6b1aaf"}
Sep 09 03:10:20.928932 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.928901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-dp6tq" event={"ID":"a5fce05f-0b08-457d-935a-b6b30a5c83fc","Type":"ContainerStarted","Data":"217dfc1d3b44c1d95b0fad4d0ee14caaab17f98c7065aeb29a027cf61ddd9579"}
Sep 09 03:10:20.929380 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.929364    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-dp6tq"
Sep 09 03:10:20.931367 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.931341    2583 generic.go:356] "Generic (PLEG): container finished" podID="db96c9f0-a549-47ee-ba38-fa42f5296d55" containerID="602400c1f6663c07fe54a45087d43868ec5ee8fb95871760913f28408e2e35b3" exitCode=0
Sep 09 03:10:20.931477 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.931428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-746hs" event={"ID":"db96c9f0-a549-47ee-ba38-fa42f5296d55","Type":"ContainerDied","Data":"602400c1f6663c07fe54a45087d43868ec5ee8fb95871760913f28408e2e35b3"}
Sep 09 03:10:20.933045 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.933019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j" event={"ID":"9c67e593-d7dc-401b-8e6a-61a92a35100e","Type":"ContainerStarted","Data":"79c61619c18328ac283730d9fe699d9064578bf545031353876bbc5ff964e223"}
Sep 09 03:10:20.934357 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.934333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg" event={"ID":"735ee480-c750-4fc9-baa2-93e5a6ae41ff","Type":"ContainerStarted","Data":"a3e27b7d6908e5e7088e158ef6c0f343e1589cbcf850efe6e8611e2d188d4679"}
Sep 09 03:10:20.942433 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.942413    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-dp6tq"
Sep 09 03:10:20.945270 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:20.945228    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-dp6tq" podStartSLOduration=1.32085972 podStartE2EDuration="18.945209808s" totalImagesPullingTime="17.624350088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:20.943525407 +0000 UTC m=+270.457607582" watchObservedRunningTime="2026-09-09 03:10:20.945209808 +0000 UTC m=+270.459291958"
Sep 09 03:10:21.944718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:21.944612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-746hs" event={"ID":"db96c9f0-a549-47ee-ba38-fa42f5296d55","Type":"ContainerStarted","Data":"62d4b5939222cc8b684b4b1afde6cc69c15e0bdda28a17f04dc66bc8d6c6164f"}
Sep 09 03:10:21.944718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:21.944673    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-746hs" event={"ID":"db96c9f0-a549-47ee-ba38-fa42f5296d55","Type":"ContainerStarted","Data":"d5c1fbd657f5c20008e549acffb12820f03578ed24f4529d69b4778fd76283d1"}
Sep 09 03:10:22.462675 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:22.462597    2583 patch_prober.go:28] interesting pod/image-registry-5dd9bdf694-zvrrf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:22.463113 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:22.462706    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf" podUID="d487509e-dff4-4996-bacb-7d83f0b599ad" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:22.815900 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:22.815786    2583 patch_prober.go:28] interesting pod/image-registry-54bf464b6f-7wmhs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:22.815900 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:22.815876    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podUID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:24.857394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:24.857347    2583 patch_prober.go:28] interesting pod/image-registry-54bf464b6f-7wmhs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:24.857849 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:24.857408    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podUID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:25.964029 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:25.963986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg" event={"ID":"735ee480-c750-4fc9-baa2-93e5a6ae41ff","Type":"ContainerStarted","Data":"827c30f137ec711420140ce86b118966d42accb8dd0e8b0319bd1f5f65d48693"}
Sep 09 03:10:25.966071 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:25.966039    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688d9c44c-hfhb4" event={"ID":"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd","Type":"ContainerStarted","Data":"94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa"}
Sep 09 03:10:25.967884 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:25.967859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j" event={"ID":"9c67e593-d7dc-401b-8e6a-61a92a35100e","Type":"ContainerStarted","Data":"430631b0a4bdb2f35f7a7c2890b1169de47f7f71f1cf79a230541e675224052e"}
Sep 09 03:10:25.968098 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:25.968077    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:25.974148 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:25.974124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j"
Sep 09 03:10:25.984297 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:25.982633    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-746hs" podStartSLOduration=6.621819439 podStartE2EDuration="14.982615832s" totalImagesPullingTime="8.267855155s" totalInitContainerRuntime="92.941238ms" isStatefulPod=true podCreationTimestamp="2026-09-09 03:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:21.967397415 +0000 UTC m=+271.481479582" watchObservedRunningTime="2026-09-09 03:10:25.982615832 +0000 UTC m=+275.496697983"
Sep 09 03:10:25.984297 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:25.983226    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-688d9c44c-hfhb4" podStartSLOduration=2.056882942 podStartE2EDuration="6.983216635s" totalImagesPullingTime="4.926333693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:25.98199733 +0000 UTC m=+275.496090565" watchObservedRunningTime="2026-09-09 03:10:25.983216635 +0000 UTC m=+275.497298788"
Sep 09 03:10:25.996081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:25.996040    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-gn28j" podStartSLOduration=6.045161744 podStartE2EDuration="10.996026914s" totalImagesPullingTime="4.95086517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:25.994879701 +0000 UTC m=+275.508961877" watchObservedRunningTime="2026-09-09 03:10:25.996026914 +0000 UTC m=+275.510109075"
Sep 09 03:10:27.476694 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.476477    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf" podUID="d487509e-dff4-4996-bacb-7d83f0b599ad" containerName="registry" containerID="cri-o://a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7" gracePeriod=30
Sep 09 03:10:27.784081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.783532    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:10:27.933472 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.933391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d487509e-dff4-4996-bacb-7d83f0b599ad-ca-trust-extracted\") pod \"d487509e-dff4-4996-bacb-7d83f0b599ad\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") "
Sep 09 03:10:27.933472 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.933430    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-installation-pull-secrets\") pod \"d487509e-dff4-4996-bacb-7d83f0b599ad\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") "
Sep 09 03:10:27.933472 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.933457    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") pod \"d487509e-dff4-4996-bacb-7d83f0b599ad\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") "
Sep 09 03:10:27.933744 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.933483    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwv9p\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-kube-api-access-bwv9p\") pod \"d487509e-dff4-4996-bacb-7d83f0b599ad\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") "
Sep 09 03:10:27.935171 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.933914    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-bound-sa-token\") pod \"d487509e-dff4-4996-bacb-7d83f0b599ad\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") "
Sep 09 03:10:27.935171 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.933996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-certificates\") pod \"d487509e-dff4-4996-bacb-7d83f0b599ad\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") "
Sep 09 03:10:27.935171 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.934034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-image-registry-private-configuration\") pod \"d487509e-dff4-4996-bacb-7d83f0b599ad\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") "
Sep 09 03:10:27.935171 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.934088    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-trusted-ca\") pod \"d487509e-dff4-4996-bacb-7d83f0b599ad\" (UID: \"d487509e-dff4-4996-bacb-7d83f0b599ad\") "
Sep 09 03:10:27.935171 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.934601    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d487509e-dff4-4996-bacb-7d83f0b599ad" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:10:27.935171 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.934868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d487509e-dff4-4996-bacb-7d83f0b599ad" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:10:27.937979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.937946    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d487509e-dff4-4996-bacb-7d83f0b599ad" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:10:27.937979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.937968    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d487509e-dff4-4996-bacb-7d83f0b599ad" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:10:27.938133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.937986    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-kube-api-access-bwv9p" (OuterVolumeSpecName: "kube-api-access-bwv9p") pod "d487509e-dff4-4996-bacb-7d83f0b599ad" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad"). InnerVolumeSpecName "kube-api-access-bwv9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:10:27.938459 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.938436    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d487509e-dff4-4996-bacb-7d83f0b599ad" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:10:27.938588 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.938563    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d487509e-dff4-4996-bacb-7d83f0b599ad" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:10:27.943821 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.943795    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d487509e-dff4-4996-bacb-7d83f0b599ad-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d487509e-dff4-4996-bacb-7d83f0b599ad" (UID: "d487509e-dff4-4996-bacb-7d83f0b599ad"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:10:27.978244 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.978211    2583 generic.go:356] "Generic (PLEG): container finished" podID="d487509e-dff4-4996-bacb-7d83f0b599ad" containerID="a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7" exitCode=0
Sep 09 03:10:27.978397 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.978318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf" event={"ID":"d487509e-dff4-4996-bacb-7d83f0b599ad","Type":"ContainerDied","Data":"a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7"}
Sep 09 03:10:27.978397 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.978334    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf"
Sep 09 03:10:27.978507 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.978410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd9bdf694-zvrrf" event={"ID":"d487509e-dff4-4996-bacb-7d83f0b599ad","Type":"ContainerDied","Data":"3d614bdf22106ae91ed3e3bfcb07e6dfa7632b4b74866de8b287d1e848bc6d72"}
Sep 09 03:10:27.978507 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.978444    2583 scope.go:117] "RemoveContainer" containerID="a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7"
Sep 09 03:10:27.982422 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.982299    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg" event={"ID":"735ee480-c750-4fc9-baa2-93e5a6ae41ff","Type":"ContainerStarted","Data":"6236416c401e414c1c7b1caf6eb726a01b2e88d1e4567d306ed54bfca8a3d0a4"}
Sep 09 03:10:27.982627 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.982599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg" event={"ID":"735ee480-c750-4fc9-baa2-93e5a6ae41ff","Type":"ContainerStarted","Data":"f9172717941d202cfe0223afba8a74b7c52c3b03d6ca4d5fa0ed8acefe66b84f"}
Sep 09 03:10:27.995871 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.995850    2583 scope.go:117] "RemoveContainer" containerID="a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7"
Sep 09 03:10:27.996272 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:10:27.996243    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7\": container with ID starting with a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7 not found: ID does not exist" containerID="a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7"
Sep 09 03:10:27.996343 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:27.996280    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7"} err="failed to get container status \"a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7\": rpc error: code = NotFound desc = could not find container \"a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7\": container with ID starting with a886500290cfde5021785908ab5b7ea8657f70dcf6b0b66566760c76da7234b7 not found: ID does not exist"
Sep 09 03:10:28.001783 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.001466    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d8c95897f-7rfjg" podStartSLOduration=5.40730155 podStartE2EDuration="12.001447522s" totalImagesPullingTime="6.594145972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:10:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:28.000013424 +0000 UTC m=+277.514095594" watchObservedRunningTime="2026-09-09 03:10:28.001447522 +0000 UTC m=+277.515529673"
Sep 09 03:10:28.016497 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.016469    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dd9bdf694-zvrrf"]
Sep 09 03:10:28.019581 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.019552    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dd9bdf694-zvrrf"]
Sep 09 03:10:28.035215 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.035172    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-bound-sa-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:10:28.035215 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.035202    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-certificates\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:10:28.035215 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.035216    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-image-registry-private-configuration\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:10:28.035467 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.035234    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d487509e-dff4-4996-bacb-7d83f0b599ad-trusted-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:10:28.035467 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.035249    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d487509e-dff4-4996-bacb-7d83f0b599ad-ca-trust-extracted\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:10:28.035467 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.035262    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d487509e-dff4-4996-bacb-7d83f0b599ad-installation-pull-secrets\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:10:28.035467 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.035275    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-registry-tls\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:10:28.035467 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:28.035288    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwv9p\" (UniqueName: \"kubernetes.io/projected/d487509e-dff4-4996-bacb-7d83f0b599ad-kube-api-access-bwv9p\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:10:29.007375 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:29.007334    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d487509e-dff4-4996-bacb-7d83f0b599ad" path="/var/lib/kubelet/pods/d487509e-dff4-4996-bacb-7d83f0b599ad/volumes"
Sep 09 03:10:30.049940 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:30.049910    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:30.050368 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:30.050187    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:10:30.051567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:30.051543    2583 patch_prober.go:28] interesting pod/console-688d9c44c-hfhb4 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Sep 09 03:10:30.051694 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:30.051588    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-688d9c44c-hfhb4" podUID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" containerName="console" probeResult="failure" output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused"
Sep 09 03:10:32.813824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:32.813791    2583 patch_prober.go:28] interesting pod/image-registry-54bf464b6f-7wmhs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:32.814289 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:32.813878    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podUID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:33.551307 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:10:33.551261    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rg75s" podUID="b72822b1-64d7-48a4-9101-4b8dcbb6ef09"
Sep 09 03:10:34.006896 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:34.006865    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rg75s"
Sep 09 03:10:34.856191 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:34.856156    2583 patch_prober.go:28] interesting pod/image-registry-54bf464b6f-7wmhs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:34.856358 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:34.856207    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podUID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:37.315967 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:37.315915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:10:37.318535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:37.318511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72822b1-64d7-48a4-9101-4b8dcbb6ef09-metrics-tls\") pod \"dns-default-rg75s\" (UID: \"b72822b1-64d7-48a4-9101-4b8dcbb6ef09\") " pod="openshift-dns/dns-default-rg75s"
Sep 09 03:10:37.610370 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:37.610286    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8s7tp\""
Sep 09 03:10:37.618312 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:37.618290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rg75s"
Sep 09 03:10:37.926274 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:37.926237    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rg75s"]
Sep 09 03:10:37.929149 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:10:37.929115    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72822b1_64d7_48a4_9101_4b8dcbb6ef09.slice/crio-bacd0a83f668ccfa7d1f4a12a3bb146a9042703fcf62a6d146fa4ff67d28acca WatchSource:0}: Error finding container bacd0a83f668ccfa7d1f4a12a3bb146a9042703fcf62a6d146fa4ff67d28acca: Status 404 returned error can't find the container with id bacd0a83f668ccfa7d1f4a12a3bb146a9042703fcf62a6d146fa4ff67d28acca
Sep 09 03:10:38.021900 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:38.021867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rg75s" event={"ID":"b72822b1-64d7-48a4-9101-4b8dcbb6ef09","Type":"ContainerStarted","Data":"bacd0a83f668ccfa7d1f4a12a3bb146a9042703fcf62a6d146fa4ff67d28acca"}
Sep 09 03:10:40.050157 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:40.050118    2583 patch_prober.go:28] interesting pod/console-688d9c44c-hfhb4 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Sep 09 03:10:40.050710 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:40.050190    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-688d9c44c-hfhb4" podUID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" containerName="console" probeResult="failure" output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused"
Sep 09 03:10:41.035992 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:41.035950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rg75s" event={"ID":"b72822b1-64d7-48a4-9101-4b8dcbb6ef09","Type":"ContainerStarted","Data":"9d369e61476a21258555dc873c0beaa0f119a9c7cd1821fff4f1a815c26d713c"}
Sep 09 03:10:41.036298 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:41.036014    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rg75s" event={"ID":"b72822b1-64d7-48a4-9101-4b8dcbb6ef09","Type":"ContainerStarted","Data":"9e04f22c1ab17131bd7683a24a748c046b462d1a14f8a35f073f0f325eea4fc7"}
Sep 09 03:10:41.036298 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:41.036084    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rg75s"
Sep 09 03:10:41.051749 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:41.051697    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rg75s" podStartSLOduration=251.726550406 podStartE2EDuration="4m14.051680482s" totalImagesPullingTime="2.325130076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:10:41.050691474 +0000 UTC m=+290.564773642" watchObservedRunningTime="2026-09-09 03:10:41.051680482 +0000 UTC m=+290.565762632"
Sep 09 03:10:42.813622 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:42.813588    2583 patch_prober.go:28] interesting pod/image-registry-54bf464b6f-7wmhs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:42.814098 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:42.813636    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podUID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:42.814098 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:42.813671    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:10:42.814098 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:42.814090    2583 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"88a66b4d651fb842eb3037c592acece2b1239b9154cc4416fba87d7506d272c3"} pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 09 03:10:42.817643 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:42.817615    2583 patch_prober.go:28] interesting pod/image-registry-54bf464b6f-7wmhs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:42.817778 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:42.817667    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podUID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:50.050563 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:50.050517    2583 patch_prober.go:28] interesting pod/console-688d9c44c-hfhb4 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Sep 09 03:10:50.050989 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:50.050589    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-688d9c44c-hfhb4" podUID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" containerName="console" probeResult="failure" output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused"
Sep 09 03:10:50.876287 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:50.876260    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:10:50.876414 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:50.876269    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:10:50.882445 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:50.882425    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:10:50.882930 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:50.882913    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:10:50.888318 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:50.888301    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 03:10:51.043405 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:51.043269    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rg75s"
Sep 09 03:10:52.818501 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:52.818466    2583 patch_prober.go:28] interesting pod/image-registry-54bf464b6f-7wmhs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:10:52.818900 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:52.818548    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podUID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:10:57.099170 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:57.099134    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a215f92-12cb-448a-b375-5a5b668a0998" containerID="8ba404be189a7b12de4ef02a0b8f67194ced9ced4907ca81dfc243df866ef331" exitCode=0
Sep 09 03:10:57.099594 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:57.099212    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749" event={"ID":"0a215f92-12cb-448a-b375-5a5b668a0998","Type":"ContainerDied","Data":"8ba404be189a7b12de4ef02a0b8f67194ced9ced4907ca81dfc243df866ef331"}
Sep 09 03:10:57.099666 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:57.099601    2583 scope.go:117] "RemoveContainer" containerID="8ba404be189a7b12de4ef02a0b8f67194ced9ced4907ca81dfc243df866ef331"
Sep 09 03:10:58.105715 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:58.105679    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-k6749" event={"ID":"0a215f92-12cb-448a-b375-5a5b668a0998","Type":"ContainerStarted","Data":"1dbe01bf2f2e0dfa05c84783d1280c6825f7acf8c438cc18b4695737342040cd"}
Sep 09 03:10:58.489042 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:10:58.489009    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-688d9c44c-hfhb4"]
Sep 09 03:11:02.818015 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:02.817988    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:11:07.835407 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:07.835363    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" podUID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerName="registry" containerID="cri-o://88a66b4d651fb842eb3037c592acece2b1239b9154cc4416fba87d7506d272c3" gracePeriod=30
Sep 09 03:11:09.149574 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:09.149485    2583 generic.go:356] "Generic (PLEG): container finished" podID="f725e7d0-9c83-4232-92d7-dab1bf647b30" containerID="88a66b4d651fb842eb3037c592acece2b1239b9154cc4416fba87d7506d272c3" exitCode=0
Sep 09 03:11:09.149953 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:09.149576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" event={"ID":"f725e7d0-9c83-4232-92d7-dab1bf647b30","Type":"ContainerDied","Data":"88a66b4d651fb842eb3037c592acece2b1239b9154cc4416fba87d7506d272c3"}
Sep 09 03:11:09.149953 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:09.149630    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:11:09.149953 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:09.149648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs" event={"ID":"f725e7d0-9c83-4232-92d7-dab1bf647b30","Type":"ContainerStarted","Data":"78b65340f97f8bd9dc9bccbd6a947dddb4510fd387919ccabfb84ddd723242b9"}
Sep 09 03:11:12.161711 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:12.161669    2583 generic.go:356] "Generic (PLEG): container finished" podID="c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1" containerID="dd447b4458960d5e9f7676e5d2e7ad4ce555f9a8c4afd95ab70dc876dd8d8f49" exitCode=0
Sep 09 03:11:12.162260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:12.161743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vt567" event={"ID":"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1","Type":"ContainerDied","Data":"dd447b4458960d5e9f7676e5d2e7ad4ce555f9a8c4afd95ab70dc876dd8d8f49"}
Sep 09 03:11:12.162260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:12.162162    2583 scope.go:117] "RemoveContainer" containerID="dd447b4458960d5e9f7676e5d2e7ad4ce555f9a8c4afd95ab70dc876dd8d8f49"
Sep 09 03:11:13.167308 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:13.167278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vt567" event={"ID":"c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1","Type":"ContainerStarted","Data":"fe0b31ac7a93ecdd523ec42f8dbbbe5e2cbc655af450675de19227d754c8cea4"}
Sep 09 03:11:13.222011 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:13.221979    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54bf464b6f-7wmhs_f725e7d0-9c83-4232-92d7-dab1bf647b30/registry/0.log"
Sep 09 03:11:13.228667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:13.228640    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54bf464b6f-7wmhs_f725e7d0-9c83-4232-92d7-dab1bf647b30/registry/1.log"
Sep 09 03:11:13.511074 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:13.510987    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2p8lk_9b96da89-37da-4ea3-a3f6-33282f05e059/node-ca/0.log"
Sep 09 03:11:14.511225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:14.511195    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dvljc_dae8f832-f923-4354-9649-b1ddc523ebf8/serve-healthcheck-canary/0.log"
Sep 09 03:11:23.509327 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.509259    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-688d9c44c-hfhb4" podUID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" containerName="console" containerID="cri-o://94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa" gracePeriod=15
Sep 09 03:11:23.760117 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.760055    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:11:23.863977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.863951    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-serving-cert\") pod \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") "
Sep 09 03:11:23.864167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864005    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cx8z\" (UniqueName: \"kubernetes.io/projected/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-kube-api-access-7cx8z\") pod \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") "
Sep 09 03:11:23.864167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864055    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-oauth-serving-cert\") pod \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") "
Sep 09 03:11:23.864167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864083    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-oauth-config\") pod \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") "
Sep 09 03:11:23.864167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-service-ca\") pod \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") "
Sep 09 03:11:23.864167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864128    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-trusted-ca-bundle\") pod \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") "
Sep 09 03:11:23.864167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864148    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-config\") pod \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\" (UID: \"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd\") "
Sep 09 03:11:23.864548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864486    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-service-ca" (OuterVolumeSpecName: "service-ca") pod "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" (UID: "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:23.864659 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864543    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" (UID: "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:23.864659 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864580    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-config" (OuterVolumeSpecName: "console-config") pod "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" (UID: "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:23.864659 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.864608    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" (UID: "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:11:23.866593 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.866563    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" (UID: "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:23.866790 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.866765    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" (UID: "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:11:23.866864 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.866803    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-kube-api-access-7cx8z" (OuterVolumeSpecName: "kube-api-access-7cx8z") pod "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" (UID: "93a0c67c-7b98-4d77-9e15-04ddc9f36cdd"). InnerVolumeSpecName "kube-api-access-7cx8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:11:23.965390 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.965355    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-trusted-ca-bundle\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:11:23.965390 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.965389    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-config\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:11:23.965632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.965404    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-serving-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:11:23.965632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.965417    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cx8z\" (UniqueName: \"kubernetes.io/projected/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-kube-api-access-7cx8z\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:11:23.965632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.965427    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-oauth-serving-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:11:23.965632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.965435    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-console-oauth-config\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:11:23.965632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:23.965443    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:11:24.211608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.211578    2583 generic.go:356] "Generic (PLEG): container finished" podID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" containerID="94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa" exitCode=0
Sep 09 03:11:24.211800 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.211623    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688d9c44c-hfhb4"
Sep 09 03:11:24.211800 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.211624    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688d9c44c-hfhb4" event={"ID":"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd","Type":"ContainerDied","Data":"94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa"}
Sep 09 03:11:24.211800 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.211744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688d9c44c-hfhb4" event={"ID":"93a0c67c-7b98-4d77-9e15-04ddc9f36cdd","Type":"ContainerDied","Data":"0659f68e9dfe05763f831581cecdfa1715014e51cf86764a19acd14a8e6b1aaf"}
Sep 09 03:11:24.211800 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.211770    2583 scope.go:117] "RemoveContainer" containerID="94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa"
Sep 09 03:11:24.224647 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.224616    2583 scope.go:117] "RemoveContainer" containerID="94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa"
Sep 09 03:11:24.225049 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:11:24.225021    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa\": container with ID starting with 94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa not found: ID does not exist" containerID="94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa"
Sep 09 03:11:24.225174 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.225061    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa"} err="failed to get container status \"94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa\": rpc error: code = NotFound desc = could not find container \"94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa\": container with ID starting with 94730d54f73559ff694781e2515010af34a7e9529d73173029e29b8dc96e8cfa not found: ID does not exist"
Sep 09 03:11:24.239094 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.239066    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-688d9c44c-hfhb4"]
Sep 09 03:11:24.241767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:24.241741    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-688d9c44c-hfhb4"]
Sep 09 03:11:25.006359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:25.006330    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" path="/var/lib/kubelet/pods/93a0c67c-7b98-4d77-9e15-04ddc9f36cdd/volumes"
Sep 09 03:11:30.157533 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:11:30.157500    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54bf464b6f-7wmhs"
Sep 09 03:12:57.639883 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.639847    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7km5q"]
Sep 09 03:12:57.640334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.640169    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d487509e-dff4-4996-bacb-7d83f0b599ad" containerName="registry"
Sep 09 03:12:57.640334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.640181    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d487509e-dff4-4996-bacb-7d83f0b599ad" containerName="registry"
Sep 09 03:12:57.640334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.640189    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" containerName="console"
Sep 09 03:12:57.640334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.640195    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" containerName="console"
Sep 09 03:12:57.640334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.640260    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a0c67c-7b98-4d77-9e15-04ddc9f36cdd" containerName="console"
Sep 09 03:12:57.640334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.640268    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d487509e-dff4-4996-bacb-7d83f0b599ad" containerName="registry"
Sep 09 03:12:57.642109 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.642093    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:12:57.645526 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.645503    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 03:12:57.645937 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.645920    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 03:12:57.646049 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.645954    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-blchj\""
Sep 09 03:12:57.653270 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.653251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7km5q"]
Sep 09 03:12:57.694876 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.694821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5a4f768-35f5-4db9-81c0-8bd50783d80e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7km5q\" (UID: \"d5a4f768-35f5-4db9-81c0-8bd50783d80e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:12:57.695019 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.694930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kr8\" (UniqueName: \"kubernetes.io/projected/d5a4f768-35f5-4db9-81c0-8bd50783d80e-kube-api-access-r4kr8\") pod \"cert-manager-webhook-6486ffd997-7km5q\" (UID: \"d5a4f768-35f5-4db9-81c0-8bd50783d80e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:12:57.796282 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.796243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kr8\" (UniqueName: \"kubernetes.io/projected/d5a4f768-35f5-4db9-81c0-8bd50783d80e-kube-api-access-r4kr8\") pod \"cert-manager-webhook-6486ffd997-7km5q\" (UID: \"d5a4f768-35f5-4db9-81c0-8bd50783d80e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:12:57.796449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.796341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5a4f768-35f5-4db9-81c0-8bd50783d80e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7km5q\" (UID: \"d5a4f768-35f5-4db9-81c0-8bd50783d80e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:12:57.812650 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.812621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kr8\" (UniqueName: \"kubernetes.io/projected/d5a4f768-35f5-4db9-81c0-8bd50783d80e-kube-api-access-r4kr8\") pod \"cert-manager-webhook-6486ffd997-7km5q\" (UID: \"d5a4f768-35f5-4db9-81c0-8bd50783d80e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:12:57.814818 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.814797    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5a4f768-35f5-4db9-81c0-8bd50783d80e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7km5q\" (UID: \"d5a4f768-35f5-4db9-81c0-8bd50783d80e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:12:57.951438 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:57.951342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:12:58.236614 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:58.236588    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7km5q"]
Sep 09 03:12:58.238957 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:12:58.238929    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5a4f768_35f5_4db9_81c0_8bd50783d80e.slice/crio-1c9d173a370d52710dc4b6b5b5efbb877f3d7847900cf958374fe25eaa1fb478 WatchSource:0}: Error finding container 1c9d173a370d52710dc4b6b5b5efbb877f3d7847900cf958374fe25eaa1fb478: Status 404 returned error can't find the container with id 1c9d173a370d52710dc4b6b5b5efbb877f3d7847900cf958374fe25eaa1fb478
Sep 09 03:12:58.240976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:58.240960    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:12:58.575018 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:12:58.574932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q" event={"ID":"d5a4f768-35f5-4db9-81c0-8bd50783d80e","Type":"ContainerStarted","Data":"1c9d173a370d52710dc4b6b5b5efbb877f3d7847900cf958374fe25eaa1fb478"}
Sep 09 03:13:02.592684 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:02.592651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q" event={"ID":"d5a4f768-35f5-4db9-81c0-8bd50783d80e","Type":"ContainerStarted","Data":"bc36348021c17394ecf5e0cdf26449b0764b9601b585cec367f5944c83071e54"}
Sep 09 03:13:02.593083 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:02.592776    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:13:02.618282 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:02.618228    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q" podStartSLOduration=1.760126739 podStartE2EDuration="5.618209854s" totalImagesPullingTime="3.858083115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:02.617549333 +0000 UTC m=+432.131631496" watchObservedRunningTime="2026-09-09 03:13:02.618209854 +0000 UTC m=+432.132292003"
Sep 09 03:13:08.599723 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:08.599693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7km5q"
Sep 09 03:13:12.047955 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.047871    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"]
Sep 09 03:13:12.051040 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.051025    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"
Sep 09 03:13:12.053537 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.053513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zzqns\""
Sep 09 03:13:12.053642 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.053543    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:13:12.054455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.054439    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 03:13:12.059471 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.059451    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"]
Sep 09 03:13:12.214228 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.214184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgwjn\" (UniqueName: \"kubernetes.io/projected/409bbb80-c5d0-4f1a-8fb9-e2497463f18d-kube-api-access-fgwjn\") pod \"openshift-lws-operator-fd8ccff4c-575vx\" (UID: \"409bbb80-c5d0-4f1a-8fb9-e2497463f18d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"
Sep 09 03:13:12.214406 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.214265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/409bbb80-c5d0-4f1a-8fb9-e2497463f18d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-575vx\" (UID: \"409bbb80-c5d0-4f1a-8fb9-e2497463f18d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"
Sep 09 03:13:12.314746 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.314667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgwjn\" (UniqueName: \"kubernetes.io/projected/409bbb80-c5d0-4f1a-8fb9-e2497463f18d-kube-api-access-fgwjn\") pod \"openshift-lws-operator-fd8ccff4c-575vx\" (UID: \"409bbb80-c5d0-4f1a-8fb9-e2497463f18d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"
Sep 09 03:13:12.314917 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.314755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/409bbb80-c5d0-4f1a-8fb9-e2497463f18d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-575vx\" (UID: \"409bbb80-c5d0-4f1a-8fb9-e2497463f18d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"
Sep 09 03:13:12.315196 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.315177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/409bbb80-c5d0-4f1a-8fb9-e2497463f18d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-575vx\" (UID: \"409bbb80-c5d0-4f1a-8fb9-e2497463f18d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"
Sep 09 03:13:12.322434 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.322407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgwjn\" (UniqueName: \"kubernetes.io/projected/409bbb80-c5d0-4f1a-8fb9-e2497463f18d-kube-api-access-fgwjn\") pod \"openshift-lws-operator-fd8ccff4c-575vx\" (UID: \"409bbb80-c5d0-4f1a-8fb9-e2497463f18d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"
Sep 09 03:13:12.360126 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.360094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"
Sep 09 03:13:12.628307 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.628175    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx"]
Sep 09 03:13:12.631097 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:13:12.631071    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod409bbb80_c5d0_4f1a_8fb9_e2497463f18d.slice/crio-8e43dd514e87fe15565680b4d3e83b3e72a61202ba5f631f8c2268a40770329e WatchSource:0}: Error finding container 8e43dd514e87fe15565680b4d3e83b3e72a61202ba5f631f8c2268a40770329e: Status 404 returned error can't find the container with id 8e43dd514e87fe15565680b4d3e83b3e72a61202ba5f631f8c2268a40770329e
Sep 09 03:13:12.636755 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:12.636732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx" event={"ID":"409bbb80-c5d0-4f1a-8fb9-e2497463f18d","Type":"ContainerStarted","Data":"8e43dd514e87fe15565680b4d3e83b3e72a61202ba5f631f8c2268a40770329e"}
Sep 09 03:13:15.653269 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:15.653228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx" event={"ID":"409bbb80-c5d0-4f1a-8fb9-e2497463f18d","Type":"ContainerStarted","Data":"931b8e996d65671601d3f87068927b309ec8d7f3eb2ff020a39ad03b10c08024"}
Sep 09 03:13:15.671979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:15.671916    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-575vx" podStartSLOduration=1.083563292 podStartE2EDuration="3.671900702s" totalImagesPullingTime="2.58833741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:15.670503845 +0000 UTC m=+445.184585997" watchObservedRunningTime="2026-09-09 03:13:15.671900702 +0000 UTC m=+445.185982852"
Sep 09 03:13:35.331612 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.331577    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"]
Sep 09 03:13:35.368217 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.368194    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"]
Sep 09 03:13:35.368367 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.368288    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.370824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.370803    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 03:13:35.370824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.370811    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 03:13:35.371031 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.370885    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7r62m\""
Sep 09 03:13:35.371031 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.370950    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 03:13:35.371185 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.371169    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 03:13:35.398966 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.398938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b292da09-4620-4521-871c-4a6e9a8dd3e8-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.399114 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.398975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nn42\" (UniqueName: \"kubernetes.io/projected/b292da09-4620-4521-871c-4a6e9a8dd3e8-kube-api-access-2nn42\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.399114 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.399000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b292da09-4620-4521-871c-4a6e9a8dd3e8-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.399114 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.399094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b292da09-4620-4521-871c-4a6e9a8dd3e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.399216 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.399120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b292da09-4620-4521-871c-4a6e9a8dd3e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.499396 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.499359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b292da09-4620-4521-871c-4a6e9a8dd3e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.499396 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.499399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b292da09-4620-4521-871c-4a6e9a8dd3e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.499641 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.499426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b292da09-4620-4521-871c-4a6e9a8dd3e8-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.499641 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.499450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nn42\" (UniqueName: \"kubernetes.io/projected/b292da09-4620-4521-871c-4a6e9a8dd3e8-kube-api-access-2nn42\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.499641 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.499485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b292da09-4620-4521-871c-4a6e9a8dd3e8-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.499978 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.499942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b292da09-4620-4521-871c-4a6e9a8dd3e8-manifests\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.502004 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.501972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b292da09-4620-4521-871c-4a6e9a8dd3e8-tmp\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.502004 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.501984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b292da09-4620-4521-871c-4a6e9a8dd3e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.502118 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.502079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b292da09-4620-4521-871c-4a6e9a8dd3e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.511595 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.511573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nn42\" (UniqueName: \"kubernetes.io/projected/b292da09-4620-4521-871c-4a6e9a8dd3e8-kube-api-access-2nn42\") pod \"opendatahub-operator-controller-manager-856874fcb4-t7cg2\" (UID: \"b292da09-4620-4521-871c-4a6e9a8dd3e8\") " pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.678532 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.678496    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:35.955404 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:35.955377    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"]
Sep 09 03:13:35.956996 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:13:35.956969    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb292da09_4620_4521_871c_4a6e9a8dd3e8.slice/crio-8ee99db1b200a82f494e71705b594c246531411b6a0897c7c05f5ffb224c9741 WatchSource:0}: Error finding container 8ee99db1b200a82f494e71705b594c246531411b6a0897c7c05f5ffb224c9741: Status 404 returned error can't find the container with id 8ee99db1b200a82f494e71705b594c246531411b6a0897c7c05f5ffb224c9741
Sep 09 03:13:36.739516 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:36.739471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2" event={"ID":"b292da09-4620-4521-871c-4a6e9a8dd3e8","Type":"ContainerStarted","Data":"8ee99db1b200a82f494e71705b594c246531411b6a0897c7c05f5ffb224c9741"}
Sep 09 03:13:39.755239 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:39.755197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2" event={"ID":"b292da09-4620-4521-871c-4a6e9a8dd3e8","Type":"ContainerStarted","Data":"712c2b520fc577820449d645244633ab8a84f9263defeccda6401b49180440db"}
Sep 09 03:13:40.760413 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:40.760376    2583 generic.go:356] "Generic (PLEG): container finished" podID="b292da09-4620-4521-871c-4a6e9a8dd3e8" containerID="712c2b520fc577820449d645244633ab8a84f9263defeccda6401b49180440db" exitCode=0
Sep 09 03:13:40.760799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:40.760457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2" event={"ID":"b292da09-4620-4521-871c-4a6e9a8dd3e8","Type":"ContainerDied","Data":"712c2b520fc577820449d645244633ab8a84f9263defeccda6401b49180440db"}
Sep 09 03:13:41.766802 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:41.766770    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2" event={"ID":"b292da09-4620-4521-871c-4a6e9a8dd3e8","Type":"ContainerStarted","Data":"e52e1a48bb7f66801761f258f4aae4b6f56a97214a72ea966b489c939b092544"}
Sep 09 03:13:41.767222 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:41.766876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:41.789291 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:41.789248    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2" podStartSLOduration=2.221507103 podStartE2EDuration="6.789233973s" totalImagesPullingTime="3.511337039s" totalInitContainerRuntime="1.056389831s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:41.787189297 +0000 UTC m=+471.301271446" watchObservedRunningTime="2026-09-09 03:13:41.789233973 +0000 UTC m=+471.303316124"
Sep 09 03:13:52.773789 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:52.773754    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-856874fcb4-t7cg2"
Sep 09 03:13:54.638612 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.638572    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"]
Sep 09 03:13:54.645889 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.645862    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.649193 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.649170    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 03:13:54.649929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.649906    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 03:13:54.650223 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.650204    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 03:13:54.651052 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.651032    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pt4b5\""
Sep 09 03:13:54.651052 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.651043    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 03:13:54.678073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.678039    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"]
Sep 09 03:13:54.749203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.749166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7m7\" (UniqueName: \"kubernetes.io/projected/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-kube-api-access-2b7m7\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.749381 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.749275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-tmp\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.749381 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.749324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-tls-certs\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.849956 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.849912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-tls-certs\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.849956 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.849954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7m7\" (UniqueName: \"kubernetes.io/projected/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-kube-api-access-2b7m7\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.850227 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.850009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-tmp\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.852577 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.852545    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-tmp\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.852727 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.852710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-tls-certs\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.858926 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.858897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7m7\" (UniqueName: \"kubernetes.io/projected/25b63d06-0e54-47c5-bfef-8afebf0b5fa3-kube-api-access-2b7m7\") pod \"kube-auth-proxy-5c8d5f99fb-794r4\" (UID: \"25b63d06-0e54-47c5-bfef-8afebf0b5fa3\") " pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:54.956346 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:54.956254    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"
Sep 09 03:13:55.228753 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:55.228726    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4"]
Sep 09 03:13:55.231279 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:13:55.231245    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b63d06_0e54_47c5_bfef_8afebf0b5fa3.slice/crio-0ef9e5927c1ea1832bee0577aa1e5b98acfcbc1d72610c2003037998da703fc6 WatchSource:0}: Error finding container 0ef9e5927c1ea1832bee0577aa1e5b98acfcbc1d72610c2003037998da703fc6: Status 404 returned error can't find the container with id 0ef9e5927c1ea1832bee0577aa1e5b98acfcbc1d72610c2003037998da703fc6
Sep 09 03:13:55.831384 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:55.831294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4" event={"ID":"25b63d06-0e54-47c5-bfef-8afebf0b5fa3","Type":"ContainerStarted","Data":"0ef9e5927c1ea1832bee0577aa1e5b98acfcbc1d72610c2003037998da703fc6"}
Sep 09 03:13:58.847383 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:58.847348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4" event={"ID":"25b63d06-0e54-47c5-bfef-8afebf0b5fa3","Type":"ContainerStarted","Data":"b7529d776ad0ce7f0d17aaab8ba28bee63b96e6086d4331444b5c7a7e4267a02"}
Sep 09 03:13:58.865498 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:13:58.865442    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c8d5f99fb-794r4" podStartSLOduration=2.148956887 podStartE2EDuration="4.865425342s" totalImagesPullingTime="2.716468455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:58.863168224 +0000 UTC m=+488.377250376" watchObservedRunningTime="2026-09-09 03:13:58.865425342 +0000 UTC m=+488.379507575"
Sep 09 03:14:02.248995 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.248953    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"]
Sep 09 03:14:02.251844 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.251810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.254475 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.254445    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 03:14:02.254648 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.254486    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-d9dzl\""
Sep 09 03:14:02.264412 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.264386    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"]
Sep 09 03:14:02.318842 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.318791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.319018 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.318863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.319018 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.318926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.319018 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.318944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.319018 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.318962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f848d1-662d-4c7a-83a8-a44975d79e72-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.319018 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.319002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2cg4\" (UniqueName: \"kubernetes.io/projected/d0f848d1-662d-4c7a-83a8-a44975d79e72-kube-api-access-n2cg4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.319203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.319043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.319203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.319063    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.319203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.319086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.419907 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.419853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.419920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.419961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.419992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f848d1-662d-4c7a-83a8-a44975d79e72-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2cg4\" (UniqueName: \"kubernetes.io/projected/d0f848d1-662d-4c7a-83a8-a44975d79e72-kube-api-access-n2cg4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420516 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420697 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420671    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420854 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420854 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.420979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.420935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f848d1-662d-4c7a-83a8-a44975d79e72-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.423042 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.423022    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.423384 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.423363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.431027 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.431004    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2cg4\" (UniqueName: \"kubernetes.io/projected/d0f848d1-662d-4c7a-83a8-a44975d79e72-kube-api-access-n2cg4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.431219 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.431200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f848d1-662d-4c7a-83a8-a44975d79e72-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn\" (UID: \"d0f848d1-662d-4c7a-83a8-a44975d79e72\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.562721 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.562624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:02.836809 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.836758    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"]
Sep 09 03:14:02.842819 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:14:02.842789    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f848d1_662d_4c7a_83a8_a44975d79e72.slice/crio-97c8d8f23134bf5d2df10c654a943a75df88c465157fe2e3b5e7b9df161e1d9f WatchSource:0}: Error finding container 97c8d8f23134bf5d2df10c654a943a75df88c465157fe2e3b5e7b9df161e1d9f: Status 404 returned error can't find the container with id 97c8d8f23134bf5d2df10c654a943a75df88c465157fe2e3b5e7b9df161e1d9f
Sep 09 03:14:02.865371 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:02.865330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn" event={"ID":"d0f848d1-662d-4c7a-83a8-a44975d79e72","Type":"ContainerStarted","Data":"97c8d8f23134bf5d2df10c654a943a75df88c465157fe2e3b5e7b9df161e1d9f"}
Sep 09 03:14:05.135719 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.135682    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-566b4"]
Sep 09 03:14:05.141150 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.141126    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:05.143705 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.143680    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 03:14:05.144087 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.144061    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xrmrk\""
Sep 09 03:14:05.147555 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.147526    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-566b4"]
Sep 09 03:14:05.246909 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.246877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d060b18-4360-4ba2-b8ac-320431078102-cert\") pod \"odh-model-controller-7998c67dc4-566b4\" (UID: \"7d060b18-4360-4ba2-b8ac-320431078102\") " pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:05.247106 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.246931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbw2\" (UniqueName: \"kubernetes.io/projected/7d060b18-4360-4ba2-b8ac-320431078102-kube-api-access-4sbw2\") pod \"odh-model-controller-7998c67dc4-566b4\" (UID: \"7d060b18-4360-4ba2-b8ac-320431078102\") " pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:05.347934 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.347896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d060b18-4360-4ba2-b8ac-320431078102-cert\") pod \"odh-model-controller-7998c67dc4-566b4\" (UID: \"7d060b18-4360-4ba2-b8ac-320431078102\") " pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:05.347934 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.347937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbw2\" (UniqueName: \"kubernetes.io/projected/7d060b18-4360-4ba2-b8ac-320431078102-kube-api-access-4sbw2\") pod \"odh-model-controller-7998c67dc4-566b4\" (UID: \"7d060b18-4360-4ba2-b8ac-320431078102\") " pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:05.348369 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:14:05.348068    2583 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 03:14:05.348369 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:14:05.348149    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d060b18-4360-4ba2-b8ac-320431078102-cert podName:7d060b18-4360-4ba2-b8ac-320431078102 nodeName:}" failed. No retries permitted until 2026-09-09 03:14:05.848127228 +0000 UTC m=+495.362209356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7d060b18-4360-4ba2-b8ac-320431078102-cert") pod "odh-model-controller-7998c67dc4-566b4" (UID: "7d060b18-4360-4ba2-b8ac-320431078102") : secret "odh-model-controller-webhook-cert" not found
Sep 09 03:14:05.357073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.357020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbw2\" (UniqueName: \"kubernetes.io/projected/7d060b18-4360-4ba2-b8ac-320431078102-kube-api-access-4sbw2\") pod \"odh-model-controller-7998c67dc4-566b4\" (UID: \"7d060b18-4360-4ba2-b8ac-320431078102\") " pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:05.494858 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.494752    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zxjpc"]
Sep 09 03:14:05.497325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.497302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:05.500067 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.500046    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 09 03:14:05.510454 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.510376    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zxjpc"]
Sep 09 03:14:05.549607 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.549573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktspd\" (UniqueName: \"kubernetes.io/projected/8977b470-c5fe-48f6-bf98-98f2c2539a1b-kube-api-access-ktspd\") pod \"model-serving-api-54d89bd5fc-zxjpc\" (UID: \"8977b470-c5fe-48f6-bf98-98f2c2539a1b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:05.549763 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.549642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8977b470-c5fe-48f6-bf98-98f2c2539a1b-tls-certs\") pod \"model-serving-api-54d89bd5fc-zxjpc\" (UID: \"8977b470-c5fe-48f6-bf98-98f2c2539a1b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:05.650823 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.650788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktspd\" (UniqueName: \"kubernetes.io/projected/8977b470-c5fe-48f6-bf98-98f2c2539a1b-kube-api-access-ktspd\") pod \"model-serving-api-54d89bd5fc-zxjpc\" (UID: \"8977b470-c5fe-48f6-bf98-98f2c2539a1b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:05.651038 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.650912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8977b470-c5fe-48f6-bf98-98f2c2539a1b-tls-certs\") pod \"model-serving-api-54d89bd5fc-zxjpc\" (UID: \"8977b470-c5fe-48f6-bf98-98f2c2539a1b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:05.653609 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.653583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8977b470-c5fe-48f6-bf98-98f2c2539a1b-tls-certs\") pod \"model-serving-api-54d89bd5fc-zxjpc\" (UID: \"8977b470-c5fe-48f6-bf98-98f2c2539a1b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:05.658737 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.658712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktspd\" (UniqueName: \"kubernetes.io/projected/8977b470-c5fe-48f6-bf98-98f2c2539a1b-kube-api-access-ktspd\") pod \"model-serving-api-54d89bd5fc-zxjpc\" (UID: \"8977b470-c5fe-48f6-bf98-98f2c2539a1b\") " pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:05.810367 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.810286    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:05.853325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.853283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d060b18-4360-4ba2-b8ac-320431078102-cert\") pod \"odh-model-controller-7998c67dc4-566b4\" (UID: \"7d060b18-4360-4ba2-b8ac-320431078102\") " pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:05.857979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:05.857952    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d060b18-4360-4ba2-b8ac-320431078102-cert\") pod \"odh-model-controller-7998c67dc4-566b4\" (UID: \"7d060b18-4360-4ba2-b8ac-320431078102\") " pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:06.057238 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:06.057207    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:06.090647 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:06.090615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zxjpc"]
Sep 09 03:14:06.969315 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:14:06.969272    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8977b470_c5fe_48f6_bf98_98f2c2539a1b.slice/crio-7596c370a2112e621be97e09cfcddd9e0e305ec107348a75b08b678b748c4fda WatchSource:0}: Error finding container 7596c370a2112e621be97e09cfcddd9e0e305ec107348a75b08b678b748c4fda: Status 404 returned error can't find the container with id 7596c370a2112e621be97e09cfcddd9e0e305ec107348a75b08b678b748c4fda
Sep 09 03:14:07.011235 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:07.011179    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:14:07.011356 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:07.011290    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:14:07.011356 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:07.011335    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:14:07.240441 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:07.240417    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-566b4"]
Sep 09 03:14:07.242772 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:14:07.242748    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d060b18_4360_4ba2_b8ac_320431078102.slice/crio-57c0e7303ebe44dfbd342463faeed5d548436b661274cbb29c1ac54d940c699d WatchSource:0}: Error finding container 57c0e7303ebe44dfbd342463faeed5d548436b661274cbb29c1ac54d940c699d: Status 404 returned error can't find the container with id 57c0e7303ebe44dfbd342463faeed5d548436b661274cbb29c1ac54d940c699d
Sep 09 03:14:07.894543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:07.894507    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-566b4" event={"ID":"7d060b18-4360-4ba2-b8ac-320431078102","Type":"ContainerStarted","Data":"57c0e7303ebe44dfbd342463faeed5d548436b661274cbb29c1ac54d940c699d"}
Sep 09 03:14:07.896483 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:07.896455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn" event={"ID":"d0f848d1-662d-4c7a-83a8-a44975d79e72","Type":"ContainerStarted","Data":"18e5b55adc03fd61bd3a2d50c24b626552727bd8781f128c21c7ce4debac2c88"}
Sep 09 03:14:07.897896 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:07.897867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc" event={"ID":"8977b470-c5fe-48f6-bf98-98f2c2539a1b","Type":"ContainerStarted","Data":"7596c370a2112e621be97e09cfcddd9e0e305ec107348a75b08b678b748c4fda"}
Sep 09 03:14:07.917209 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:07.917150    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn" podStartSLOduration=1.751246375 podStartE2EDuration="5.917129888s" totalImagesPullingTime="4.165883513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:14:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:07.915080706 +0000 UTC m=+497.429162876" watchObservedRunningTime="2026-09-09 03:14:07.917129888 +0000 UTC m=+497.431212038"
Sep 09 03:14:08.563999 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:08.563872    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:08.570040 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:08.570014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:08.905007 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:08.904961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc" event={"ID":"8977b470-c5fe-48f6-bf98-98f2c2539a1b","Type":"ContainerStarted","Data":"2676cfbb4e97ceb6ebeea177062bd9c6a319566c3e6a693467de6c4b3e5d346e"}
Sep 09 03:14:08.905201 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:08.905096    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:08.905378 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:08.905350    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:08.906269 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:08.906245    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn"
Sep 09 03:14:08.920497 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:08.920447    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc" podStartSLOduration=2.118292078 podStartE2EDuration="3.920426694s" totalImagesPullingTime="1.802134616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:08.919501463 +0000 UTC m=+498.433583607" watchObservedRunningTime="2026-09-09 03:14:08.920426694 +0000 UTC m=+498.434508837"
Sep 09 03:14:10.916154 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:10.916117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-566b4" event={"ID":"7d060b18-4360-4ba2-b8ac-320431078102","Type":"ContainerStarted","Data":"c10d11dbe84a8d06aa0a9971970cd0487a64e68f8524fb18b0147f954f8d3ab1"}
Sep 09 03:14:10.916512 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:10.916369    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:10.932082 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:10.932035    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-566b4" podStartSLOduration=2.823645766 podStartE2EDuration="5.932020316s" totalImagesPullingTime="3.10837455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:10.930729325 +0000 UTC m=+500.444811509" watchObservedRunningTime="2026-09-09 03:14:10.932020316 +0000 UTC m=+500.446102466"
Sep 09 03:14:14.478500 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.478448    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"]
Sep 09 03:14:14.485480 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.485456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.488908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.488884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d7dnd\""
Sep 09 03:14:14.489040 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.488890    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 03:14:14.489040 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.488928    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 03:14:14.489135 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.489044    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 03:14:14.495326 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.495301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"]
Sep 09 03:14:14.637258 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.637218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzggb\" (UniqueName: \"kubernetes.io/projected/8018038d-bd20-425b-9b38-ed0a2abd8554-kube-api-access-tzggb\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.637258 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.637259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8018038d-bd20-425b-9b38-ed0a2abd8554-metrics-cert\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.637470 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.637304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8018038d-bd20-425b-9b38-ed0a2abd8554-manager-config\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.637470 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.637354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8018038d-bd20-425b-9b38-ed0a2abd8554-cert\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.738500 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.738413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzggb\" (UniqueName: \"kubernetes.io/projected/8018038d-bd20-425b-9b38-ed0a2abd8554-kube-api-access-tzggb\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.738500 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.738457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8018038d-bd20-425b-9b38-ed0a2abd8554-metrics-cert\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.738712 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.738515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8018038d-bd20-425b-9b38-ed0a2abd8554-manager-config\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.738712 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.738540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8018038d-bd20-425b-9b38-ed0a2abd8554-cert\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.739231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.739203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8018038d-bd20-425b-9b38-ed0a2abd8554-manager-config\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.741360 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.741332    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8018038d-bd20-425b-9b38-ed0a2abd8554-cert\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.741360 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.741350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8018038d-bd20-425b-9b38-ed0a2abd8554-metrics-cert\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.749410 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.749384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzggb\" (UniqueName: \"kubernetes.io/projected/8018038d-bd20-425b-9b38-ed0a2abd8554-kube-api-access-tzggb\") pod \"lws-controller-manager-6f6f9bc9b9-xp24w\" (UID: \"8018038d-bd20-425b-9b38-ed0a2abd8554\") " pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:14.795601 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:14.795571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:15.074741 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:15.074711    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"]
Sep 09 03:14:15.079171 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:14:15.079144    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8018038d_bd20_425b_9b38_ed0a2abd8554.slice/crio-2d4033a55eef2c1cd0c68f7eba7ea67edb251c244cea733b9ff79b258518b885 WatchSource:0}: Error finding container 2d4033a55eef2c1cd0c68f7eba7ea67edb251c244cea733b9ff79b258518b885: Status 404 returned error can't find the container with id 2d4033a55eef2c1cd0c68f7eba7ea67edb251c244cea733b9ff79b258518b885
Sep 09 03:14:15.941056 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:15.941012    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w" event={"ID":"8018038d-bd20-425b-9b38-ed0a2abd8554","Type":"ContainerStarted","Data":"2d4033a55eef2c1cd0c68f7eba7ea67edb251c244cea733b9ff79b258518b885"}
Sep 09 03:14:16.948290 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:16.948199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w" event={"ID":"8018038d-bd20-425b-9b38-ed0a2abd8554","Type":"ContainerStarted","Data":"4c87d0603c979ea15118e7ef8527112bc17492f55ca08251a3ad6b1a75d6606f"}
Sep 09 03:14:16.948290 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:16.948267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:14:16.967199 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:16.967148    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w" podStartSLOduration=1.395438595 podStartE2EDuration="2.9671281s" totalImagesPullingTime="1.571689505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:16.965672718 +0000 UTC m=+506.479754881" watchObservedRunningTime="2026-09-09 03:14:16.9671281 +0000 UTC m=+506.481210250"
Sep 09 03:14:19.914160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:19.914125    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zxjpc"
Sep 09 03:14:21.923752 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:21.923716    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-566b4"
Sep 09 03:14:27.955424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:14:27.955393    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f6f9bc9b9-xp24w"
Sep 09 03:15:50.904378 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:15:50.904347    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:15:50.905284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:15:50.905258    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:15:50.914235 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:15:50.914214    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:15:50.915256 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:15:50.915236    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:16:17.538787 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.538692    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vhkq7"]
Sep 09 03:16:17.542077 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.542059    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7"
Sep 09 03:16:17.544705 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.544672    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 03:16:17.544705 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.544683    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 03:16:17.544705 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.544682    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l5zrp\""
Sep 09 03:16:17.550525 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.550504    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vhkq7"]
Sep 09 03:16:17.709881 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.709820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2k4\" (UniqueName: \"kubernetes.io/projected/511ca027-db14-4829-b254-0105ed7401f8-kube-api-access-cj2k4\") pod \"authorino-7bcfd8f9cd-vhkq7\" (UID: \"511ca027-db14-4829-b254-0105ed7401f8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7"
Sep 09 03:16:17.811125 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.811037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2k4\" (UniqueName: \"kubernetes.io/projected/511ca027-db14-4829-b254-0105ed7401f8-kube-api-access-cj2k4\") pod \"authorino-7bcfd8f9cd-vhkq7\" (UID: \"511ca027-db14-4829-b254-0105ed7401f8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7"
Sep 09 03:16:17.818887 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.818857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2k4\" (UniqueName: \"kubernetes.io/projected/511ca027-db14-4829-b254-0105ed7401f8-kube-api-access-cj2k4\") pod \"authorino-7bcfd8f9cd-vhkq7\" (UID: \"511ca027-db14-4829-b254-0105ed7401f8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7"
Sep 09 03:16:17.851595 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:17.851557    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7"
Sep 09 03:16:18.120393 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:18.120300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vhkq7"]
Sep 09 03:16:18.123041 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:16:18.123014    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511ca027_db14_4829_b254_0105ed7401f8.slice/crio-9c1ca9896454eb5c81153489123a37b5c061ec9621c01dc6a4773374581400ca WatchSource:0}: Error finding container 9c1ca9896454eb5c81153489123a37b5c061ec9621c01dc6a4773374581400ca: Status 404 returned error can't find the container with id 9c1ca9896454eb5c81153489123a37b5c061ec9621c01dc6a4773374581400ca
Sep 09 03:16:18.487546 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:18.487460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7" event={"ID":"511ca027-db14-4829-b254-0105ed7401f8","Type":"ContainerStarted","Data":"9c1ca9896454eb5c81153489123a37b5c061ec9621c01dc6a4773374581400ca"}
Sep 09 03:16:20.711894 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:20.711851    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vhkq7"]
Sep 09 03:16:21.506360 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:21.506316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7" event={"ID":"511ca027-db14-4829-b254-0105ed7401f8","Type":"ContainerStarted","Data":"b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c"}
Sep 09 03:16:21.506554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:21.506425    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7" podUID="511ca027-db14-4829-b254-0105ed7401f8" containerName="authorino" containerID="cri-o://b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c" gracePeriod=30
Sep 09 03:16:21.520959 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:21.520917    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7" podStartSLOduration=1.263782821 podStartE2EDuration="4.520900481s" totalImagesPullingTime="3.25711766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:21.51965477 +0000 UTC m=+631.033736935" watchObservedRunningTime="2026-09-09 03:16:21.520900481 +0000 UTC m=+631.034982630"
Sep 09 03:16:21.753974 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:21.753950    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7"
Sep 09 03:16:21.847689 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:21.847661    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj2k4\" (UniqueName: \"kubernetes.io/projected/511ca027-db14-4829-b254-0105ed7401f8-kube-api-access-cj2k4\") pod \"511ca027-db14-4829-b254-0105ed7401f8\" (UID: \"511ca027-db14-4829-b254-0105ed7401f8\") "
Sep 09 03:16:21.850037 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:21.850004    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511ca027-db14-4829-b254-0105ed7401f8-kube-api-access-cj2k4" (OuterVolumeSpecName: "kube-api-access-cj2k4") pod "511ca027-db14-4829-b254-0105ed7401f8" (UID: "511ca027-db14-4829-b254-0105ed7401f8"). InnerVolumeSpecName "kube-api-access-cj2k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:16:21.948401 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:21.948318    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2k4\" (UniqueName: \"kubernetes.io/projected/511ca027-db14-4829-b254-0105ed7401f8-kube-api-access-cj2k4\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:16:22.512658 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.512626    2583 generic.go:356] "Generic (PLEG): container finished" podID="511ca027-db14-4829-b254-0105ed7401f8" containerID="b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c" exitCode=0
Sep 09 03:16:22.512858 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.512671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7" event={"ID":"511ca027-db14-4829-b254-0105ed7401f8","Type":"ContainerDied","Data":"b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c"}
Sep 09 03:16:22.512858 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.512697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7" event={"ID":"511ca027-db14-4829-b254-0105ed7401f8","Type":"ContainerDied","Data":"9c1ca9896454eb5c81153489123a37b5c061ec9621c01dc6a4773374581400ca"}
Sep 09 03:16:22.512858 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.512673    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vhkq7"
Sep 09 03:16:22.512858 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.512712    2583 scope.go:117] "RemoveContainer" containerID="b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c"
Sep 09 03:16:22.521454 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.521439    2583 scope.go:117] "RemoveContainer" containerID="b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c"
Sep 09 03:16:22.521709 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:16:22.521691    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c\": container with ID starting with b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c not found: ID does not exist" containerID="b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c"
Sep 09 03:16:22.521754 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.521719    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c"} err="failed to get container status \"b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c\": rpc error: code = NotFound desc = could not find container \"b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c\": container with ID starting with b3629f40620a0f6165f1b1404570f0d32644aaf194b62f52d6570bf82ed2f29c not found: ID does not exist"
Sep 09 03:16:22.531779 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.531755    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vhkq7"]
Sep 09 03:16:22.533897 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:22.533876    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vhkq7"]
Sep 09 03:16:23.007135 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:23.007102    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511ca027-db14-4829-b254-0105ed7401f8" path="/var/lib/kubelet/pods/511ca027-db14-4829-b254-0105ed7401f8/volumes"
Sep 09 03:16:50.953875 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.953845    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-769bf8db75-tf7qf"]
Sep 09 03:16:50.954260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.954181    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511ca027-db14-4829-b254-0105ed7401f8" containerName="authorino"
Sep 09 03:16:50.954260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.954192    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="511ca027-db14-4829-b254-0105ed7401f8" containerName="authorino"
Sep 09 03:16:50.954335 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.954266    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="511ca027-db14-4829-b254-0105ed7401f8" containerName="authorino"
Sep 09 03:16:50.958554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.958536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:50.961201 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.961177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 03:16:50.961301 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.961277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 03:16:50.962250 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.962228    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-92449\""
Sep 09 03:16:50.962340 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.962232    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 03:16:50.967125 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:50.967102    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-769bf8db75-tf7qf"]
Sep 09 03:16:51.087921 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.087879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.087921 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.087918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/df7e3b9d-5507-415f-b086-02d16109fa4f-kube-api-access-4wtn6\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.088177 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.088159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-metrics-certs\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.108500 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.108469    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66bcff4f74-vs7gh"]
Sep 09 03:16:51.111710 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.111695    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.121184 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.121160    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-vs7gh"]
Sep 09 03:16:51.189211 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.189101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.189211 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.189143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/df7e3b9d-5507-415f-b086-02d16109fa4f-kube-api-access-4wtn6\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.189450 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:16:51.189261    2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 09 03:16:51.189450 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.189281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzs7\" (UniqueName: \"kubernetes.io/projected/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-kube-api-access-bmzs7\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.189450 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:16:51.189331    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert podName:df7e3b9d-5507-415f-b086-02d16109fa4f nodeName:}" failed. No retries permitted until 2026-09-09 03:16:51.689311411 +0000 UTC m=+661.203393539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert") pod "maas-controller-769bf8db75-tf7qf" (UID: "df7e3b9d-5507-415f-b086-02d16109fa4f") : secret "maas-controller-webhook-cert" not found
Sep 09 03:16:51.189450 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.189445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-metrics-certs\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.189646 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.189495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-webhook-cert\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.189646 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.189510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-metrics-certs\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.192060 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.192043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-metrics-certs\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.197382 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.197353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/df7e3b9d-5507-415f-b086-02d16109fa4f-kube-api-access-4wtn6\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.290923 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.290884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzs7\" (UniqueName: \"kubernetes.io/projected/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-kube-api-access-bmzs7\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.291111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.290964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-webhook-cert\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.291111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.290981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-metrics-certs\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.293615 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.293587    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-webhook-cert\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.293722 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.293648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-metrics-certs\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.298554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.298533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzs7\" (UniqueName: \"kubernetes.io/projected/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-kube-api-access-bmzs7\") pod \"maas-controller-66bcff4f74-vs7gh\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") " pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.422171 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.422129    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:51.694202 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.694171    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-vs7gh"]
Sep 09 03:16:51.694791 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.694767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.696859 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:16:51.696816    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3da7eac_523d_4d83_8da7_8cbf9b5e583a.slice/crio-74dabfcb998f4fed4bde99bca9257e354ba353121282ed59517a6a7be4009aae WatchSource:0}: Error finding container 74dabfcb998f4fed4bde99bca9257e354ba353121282ed59517a6a7be4009aae: Status 404 returned error can't find the container with id 74dabfcb998f4fed4bde99bca9257e354ba353121282ed59517a6a7be4009aae
Sep 09 03:16:51.697375 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.697360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert\") pod \"maas-controller-769bf8db75-tf7qf\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") " pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:51.870537 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:51.870500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:52.141354 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:52.141332    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-769bf8db75-tf7qf"]
Sep 09 03:16:52.143750 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:16:52.143723    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7e3b9d_5507_415f_b086_02d16109fa4f.slice/crio-9f050e33b8be7ba7ca5dcb2164498ea2ac92f9b12e8d92eb44c1edab8cac1467 WatchSource:0}: Error finding container 9f050e33b8be7ba7ca5dcb2164498ea2ac92f9b12e8d92eb44c1edab8cac1467: Status 404 returned error can't find the container with id 9f050e33b8be7ba7ca5dcb2164498ea2ac92f9b12e8d92eb44c1edab8cac1467
Sep 09 03:16:52.653237 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:52.653158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-vs7gh" event={"ID":"b3da7eac-523d-4d83-8da7-8cbf9b5e583a","Type":"ContainerStarted","Data":"74dabfcb998f4fed4bde99bca9257e354ba353121282ed59517a6a7be4009aae"}
Sep 09 03:16:52.654897 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:52.654802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-769bf8db75-tf7qf" event={"ID":"df7e3b9d-5507-415f-b086-02d16109fa4f","Type":"ContainerStarted","Data":"9f050e33b8be7ba7ca5dcb2164498ea2ac92f9b12e8d92eb44c1edab8cac1467"}
Sep 09 03:16:54.666667 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:54.666627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-vs7gh" event={"ID":"b3da7eac-523d-4d83-8da7-8cbf9b5e583a","Type":"ContainerStarted","Data":"d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a"}
Sep 09 03:16:54.667154 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:54.666748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:16:54.668291 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:54.668267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-769bf8db75-tf7qf" event={"ID":"df7e3b9d-5507-415f-b086-02d16109fa4f","Type":"ContainerStarted","Data":"522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0"}
Sep 09 03:16:54.668423 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:54.668393    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:16:54.683996 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:54.683946    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66bcff4f74-vs7gh" podStartSLOduration=1.600450441 podStartE2EDuration="3.683930778s" totalImagesPullingTime="2.083480337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:54.682354505 +0000 UTC m=+664.196436675" watchObservedRunningTime="2026-09-09 03:16:54.683930778 +0000 UTC m=+664.198012928"
Sep 09 03:16:54.699804 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:16:54.699748    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-769bf8db75-tf7qf" podStartSLOduration=3.066535722 podStartE2EDuration="4.69973122s" totalImagesPullingTime="1.633195498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:16:54.698312799 +0000 UTC m=+664.212394973" watchObservedRunningTime="2026-09-09 03:16:54.69973122 +0000 UTC m=+664.213813370"
Sep 09 03:17:05.675599 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:05.675562    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:17:05.676124 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:05.675674    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:17:05.734649 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:05.734613    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-769bf8db75-tf7qf"]
Sep 09 03:17:05.734912 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:05.734867    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-769bf8db75-tf7qf" podUID="df7e3b9d-5507-415f-b086-02d16109fa4f" containerName="manager" containerID="cri-o://522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0" gracePeriod=30
Sep 09 03:17:05.979231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:05.979206    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:17:06.117941 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.117907    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/df7e3b9d-5507-415f-b086-02d16109fa4f-kube-api-access-4wtn6\") pod \"df7e3b9d-5507-415f-b086-02d16109fa4f\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") "
Sep 09 03:17:06.118105 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.117987    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-metrics-certs\") pod \"df7e3b9d-5507-415f-b086-02d16109fa4f\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") "
Sep 09 03:17:06.118105 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.118034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert\") pod \"df7e3b9d-5507-415f-b086-02d16109fa4f\" (UID: \"df7e3b9d-5507-415f-b086-02d16109fa4f\") "
Sep 09 03:17:06.120264 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.120225    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df7e3b9d-5507-415f-b086-02d16109fa4f" (UID: "df7e3b9d-5507-415f-b086-02d16109fa4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:17:06.120264 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.120258    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7e3b9d-5507-415f-b086-02d16109fa4f-kube-api-access-4wtn6" (OuterVolumeSpecName: "kube-api-access-4wtn6") pod "df7e3b9d-5507-415f-b086-02d16109fa4f" (UID: "df7e3b9d-5507-415f-b086-02d16109fa4f"). InnerVolumeSpecName "kube-api-access-4wtn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:17:06.120415 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.120303    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "df7e3b9d-5507-415f-b086-02d16109fa4f" (UID: "df7e3b9d-5507-415f-b086-02d16109fa4f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:17:06.219238 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.219147    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-metrics-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:17:06.219238 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.219179    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df7e3b9d-5507-415f-b086-02d16109fa4f-webhook-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:17:06.219238 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.219190    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/df7e3b9d-5507-415f-b086-02d16109fa4f-kube-api-access-4wtn6\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:17:06.727102 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.727064    2583 generic.go:356] "Generic (PLEG): container finished" podID="df7e3b9d-5507-415f-b086-02d16109fa4f" containerID="522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0" exitCode=0
Sep 09 03:17:06.727568 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.727123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-769bf8db75-tf7qf" event={"ID":"df7e3b9d-5507-415f-b086-02d16109fa4f","Type":"ContainerDied","Data":"522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0"}
Sep 09 03:17:06.727568 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.727126    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-769bf8db75-tf7qf"
Sep 09 03:17:06.727568 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.727214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-769bf8db75-tf7qf" event={"ID":"df7e3b9d-5507-415f-b086-02d16109fa4f","Type":"ContainerDied","Data":"9f050e33b8be7ba7ca5dcb2164498ea2ac92f9b12e8d92eb44c1edab8cac1467"}
Sep 09 03:17:06.727568 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.727250    2583 scope.go:117] "RemoveContainer" containerID="522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0"
Sep 09 03:17:06.736811 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.736646    2583 scope.go:117] "RemoveContainer" containerID="522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0"
Sep 09 03:17:06.736993 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:17:06.736971    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0\": container with ID starting with 522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0 not found: ID does not exist" containerID="522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0"
Sep 09 03:17:06.737060 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.737007    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0"} err="failed to get container status \"522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0\": rpc error: code = NotFound desc = could not find container \"522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0\": container with ID starting with 522a8b34f1b6b28f9354c278b518e93eb340c6d091c655691756b11221f24ee0 not found: ID does not exist"
Sep 09 03:17:06.747110 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.747084    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-769bf8db75-tf7qf"]
Sep 09 03:17:06.753444 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:06.753419    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-769bf8db75-tf7qf"]
Sep 09 03:17:07.007779 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:07.007699    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df7e3b9d-5507-415f-b086-02d16109fa4f" path="/var/lib/kubelet/pods/df7e3b9d-5507-415f-b086-02d16109fa4f/volumes"
Sep 09 03:17:49.800434 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.800397    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"]
Sep 09 03:17:49.800897 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.800778    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df7e3b9d-5507-415f-b086-02d16109fa4f" containerName="manager"
Sep 09 03:17:49.800897 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.800790    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="df7e3b9d-5507-415f-b086-02d16109fa4f" containerName="manager"
Sep 09 03:17:49.800897 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.800875    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="df7e3b9d-5507-415f-b086-02d16109fa4f" containerName="manager"
Sep 09 03:17:49.803626 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.803609    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:49.807014 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.806989    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 03:17:49.807014 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.807001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 03:17:49.807206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.807001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vxktl\""
Sep 09 03:17:49.807206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.807144    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 03:17:49.817517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.817491    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"]
Sep 09 03:17:49.905173 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.905140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:49.905348 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.905194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:49.905348 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.905247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:49.905348 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.905279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:49.905463 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.905380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73e19ef-9123-4c0c-801d-d0b287c54767-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:49.905463 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:49.905427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdcr\" (UniqueName: \"kubernetes.io/projected/b73e19ef-9123-4c0c-801d-d0b287c54767-kube-api-access-kqdcr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006136 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006303 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006303 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006303 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006303 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73e19ef-9123-4c0c-801d-d0b287c54767-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006473 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdcr\" (UniqueName: \"kubernetes.io/projected/b73e19ef-9123-4c0c-801d-d0b287c54767-kube-api-access-kqdcr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006519 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006565    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.006707 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.006635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.008637 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.008609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73e19ef-9123-4c0c-801d-d0b287c54767-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.009035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.009014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73e19ef-9123-4c0c-801d-d0b287c54767-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.014303 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.014280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdcr\" (UniqueName: \"kubernetes.io/projected/b73e19ef-9123-4c0c-801d-d0b287c54767-kube-api-access-kqdcr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5\" (UID: \"b73e19ef-9123-4c0c-801d-d0b287c54767\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.115085 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.115007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:17:50.386950 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.386794    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"]
Sep 09 03:17:50.389319 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:17:50.389292    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73e19ef_9123_4c0c_801d_d0b287c54767.slice/crio-8d4576e373d11489ee8313bc49167e2b53cdf58efd58c0d466332b9df70bee7a WatchSource:0}: Error finding container 8d4576e373d11489ee8313bc49167e2b53cdf58efd58c0d466332b9df70bee7a: Status 404 returned error can't find the container with id 8d4576e373d11489ee8313bc49167e2b53cdf58efd58c0d466332b9df70bee7a
Sep 09 03:17:50.932936 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:50.932860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5" event={"ID":"b73e19ef-9123-4c0c-801d-d0b287c54767","Type":"ContainerStarted","Data":"8d4576e373d11489ee8313bc49167e2b53cdf58efd58c0d466332b9df70bee7a"}
Sep 09 03:17:51.938758 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:51.938717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5" event={"ID":"b73e19ef-9123-4c0c-801d-d0b287c54767","Type":"ContainerStarted","Data":"434708db085762f0570d6998e776a045e8b9e5bed4aaa4470cd6d1c954d32b78"}
Sep 09 03:17:51.956132 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:17:51.956083    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5" podStartSLOduration=1.8808770670000001 podStartE2EDuration="2.956068338s" totalImagesPullingTime="1.075191271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:51.955390571 +0000 UTC m=+721.469472735" watchObservedRunningTime="2026-09-09 03:17:51.956068338 +0000 UTC m=+721.470150488"
Sep 09 03:18:00.115775 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:00.115737    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:18:00.116224 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:00.115850    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:18:00.128615 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:00.128589    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:18:00.991039 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:00.991003    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5"
Sep 09 03:18:04.196095 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.196062    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"]
Sep 09 03:18:04.198631 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.198610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.200860 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.200817    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 03:18:04.207628 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.207602    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"]
Sep 09 03:18:04.227103 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.221220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.227103 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.221312    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.227103 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.221350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.227103 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.221441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.227103 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.221528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lfc\" (UniqueName: \"kubernetes.io/projected/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-kube-api-access-w2lfc\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.227103 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.221575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.322646 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.322587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lfc\" (UniqueName: \"kubernetes.io/projected/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-kube-api-access-w2lfc\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.322879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.322670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.322879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.322707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.322879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.322738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.322879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.322762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.322879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.322820    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.323212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.323188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.323212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.323205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.323307 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.323230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.325321 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.325298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.325511 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.325491    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.330012 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.329990    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lfc\" (UniqueName: \"kubernetes.io/projected/ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5-kube-api-access-w2lfc\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs\" (UID: \"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.509355 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.509257    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:04.779938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.779775    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"]
Sep 09 03:18:04.782669 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:18:04.782641    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded74d6f5_7ce8_44f5_a8e9_eb6ae3e51dd5.slice/crio-ccc53a07daf6fc843f0668b77075a95513c496086d92e093885caffa72f29c55 WatchSource:0}: Error finding container ccc53a07daf6fc843f0668b77075a95513c496086d92e093885caffa72f29c55: Status 404 returned error can't find the container with id ccc53a07daf6fc843f0668b77075a95513c496086d92e093885caffa72f29c55
Sep 09 03:18:04.784491 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.784473    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:18:04.998870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:04.998796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs" event={"ID":"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5","Type":"ContainerStarted","Data":"ccc53a07daf6fc843f0668b77075a95513c496086d92e093885caffa72f29c55"}
Sep 09 03:18:06.006046 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:06.006015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs" event={"ID":"ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5","Type":"ContainerStarted","Data":"b659db28a2c8ed2bd8b7e588713980b54d19c58e8a121ccf3cc3047c56daa0e6"}
Sep 09 03:18:06.042628 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:06.042571    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs" podStartSLOduration=1.86817456 podStartE2EDuration="2.042554952s" totalImagesPullingTime="174.380392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:06.041128745 +0000 UTC m=+735.555210917" watchObservedRunningTime="2026-09-09 03:18:06.042554952 +0000 UTC m=+735.556637101"
Sep 09 03:18:14.510230 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:14.510169    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:14.510710 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:14.510263    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:14.522971 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:14.522939    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:15.056256 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:15.056226    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs"
Sep 09 03:18:20.199752 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.199718    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"]
Sep 09 03:18:20.203487 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.203404    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.205910 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.205886    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 03:18:20.215355 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.215304    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"]
Sep 09 03:18:20.263366 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.263327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.263366 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.263366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.263571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.263391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.263571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.263512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d44558e4-6f32-497e-afb3-e79a58277da2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.263571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.263544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw65l\" (UniqueName: \"kubernetes.io/projected/d44558e4-6f32-497e-afb3-e79a58277da2-kube-api-access-mw65l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.263571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.263567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364363 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364363 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364607 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364607 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d44558e4-6f32-497e-afb3-e79a58277da2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364607 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw65l\" (UniqueName: \"kubernetes.io/projected/d44558e4-6f32-497e-afb3-e79a58277da2-kube-api-access-mw65l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364607 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364810 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364785    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364912 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364820    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.364912 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.364879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.367030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.367006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d44558e4-6f32-497e-afb3-e79a58277da2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.367213 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.367193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d44558e4-6f32-497e-afb3-e79a58277da2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.371766 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.371748    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw65l\" (UniqueName: \"kubernetes.io/projected/d44558e4-6f32-497e-afb3-e79a58277da2-kube-api-access-mw65l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8xs9k\" (UID: \"d44558e4-6f32-497e-afb3-e79a58277da2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.520303 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.520212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:20.792244 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:20.792162    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"]
Sep 09 03:18:20.794891 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:18:20.794859    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44558e4_6f32_497e_afb3_e79a58277da2.slice/crio-6cfc65669fe430970ed19e5e8dc51ac72d55ea8d4787c8c39bdb72f9ada466c9 WatchSource:0}: Error finding container 6cfc65669fe430970ed19e5e8dc51ac72d55ea8d4787c8c39bdb72f9ada466c9: Status 404 returned error can't find the container with id 6cfc65669fe430970ed19e5e8dc51ac72d55ea8d4787c8c39bdb72f9ada466c9
Sep 09 03:18:21.075047 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:21.074960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k" event={"ID":"d44558e4-6f32-497e-afb3-e79a58277da2","Type":"ContainerStarted","Data":"20e5021cd79d97676abf42cf872b5002e0e1ba3fece525b0b77505fde96d619e"}
Sep 09 03:18:21.075047 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:21.075015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k" event={"ID":"d44558e4-6f32-497e-afb3-e79a58277da2","Type":"ContainerStarted","Data":"6cfc65669fe430970ed19e5e8dc51ac72d55ea8d4787c8c39bdb72f9ada466c9"}
Sep 09 03:18:21.091590 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:21.091534    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k" podStartSLOduration=0.915589834 podStartE2EDuration="1.091516432s" totalImagesPullingTime="175.926598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:21.090062839 +0000 UTC m=+750.604144990" watchObservedRunningTime="2026-09-09 03:18:21.091516432 +0000 UTC m=+750.605598582"
Sep 09 03:18:30.520332 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:30.520299    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:30.520723 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:30.520351    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:30.533334 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:30.533309    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:31.135098 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:31.135070    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8xs9k"
Sep 09 03:18:55.498614 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.498584    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"]
Sep 09 03:18:55.502182 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.502163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.504425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.504400    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 03:18:55.509803 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.509782    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"]
Sep 09 03:18:55.651144 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.651109    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.651282 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.651156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.651282 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.651174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.651282 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.651259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f982dadf-881e-4d85-b5fe-88e90bc22a0b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.651383 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.651294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r228\" (UniqueName: \"kubernetes.io/projected/f982dadf-881e-4d85-b5fe-88e90bc22a0b-kube-api-access-9r228\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.651383 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.651339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.752595 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.752525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.752595 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.752567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.752595 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.752586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.752857 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.752717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f982dadf-881e-4d85-b5fe-88e90bc22a0b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.752857 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.752753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r228\" (UniqueName: \"kubernetes.io/projected/f982dadf-881e-4d85-b5fe-88e90bc22a0b-kube-api-access-9r228\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.752857 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.752804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.753045 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.753024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.753099 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.753051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.753152 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.753132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.754951 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.754925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f982dadf-881e-4d85-b5fe-88e90bc22a0b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.755194 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.755178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f982dadf-881e-4d85-b5fe-88e90bc22a0b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.759866 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.759848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r228\" (UniqueName: \"kubernetes.io/projected/f982dadf-881e-4d85-b5fe-88e90bc22a0b-kube-api-access-9r228\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mvldf\" (UID: \"f982dadf-881e-4d85-b5fe-88e90bc22a0b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:55.811732 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:55.811709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:18:56.082089 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:56.082065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"]
Sep 09 03:18:56.083789 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:18:56.083760    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf982dadf_881e_4d85_b5fe_88e90bc22a0b.slice/crio-62a52a54cfb716988107806c23e6ce790f2ce77ca52f3613eb0c2c800fe29b5e WatchSource:0}: Error finding container 62a52a54cfb716988107806c23e6ce790f2ce77ca52f3613eb0c2c800fe29b5e: Status 404 returned error can't find the container with id 62a52a54cfb716988107806c23e6ce790f2ce77ca52f3613eb0c2c800fe29b5e
Sep 09 03:18:56.241209 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:56.241182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf" event={"ID":"f982dadf-881e-4d85-b5fe-88e90bc22a0b","Type":"ContainerStarted","Data":"62a52a54cfb716988107806c23e6ce790f2ce77ca52f3613eb0c2c800fe29b5e"}
Sep 09 03:18:57.247543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:57.247509    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf" event={"ID":"f982dadf-881e-4d85-b5fe-88e90bc22a0b","Type":"ContainerStarted","Data":"501d9417fbec95762734d94bc71b42eca8b7326d1938eef4cff222a6cded554e"}
Sep 09 03:18:57.267470 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:18:57.267423    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf" podStartSLOduration=2.091914332 podStartE2EDuration="2.267409027s" totalImagesPullingTime="175.494695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:57.265504119 +0000 UTC m=+786.779586294" watchObservedRunningTime="2026-09-09 03:18:57.267409027 +0000 UTC m=+786.781491177"
Sep 09 03:19:05.812011 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:05.811975    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:19:05.812398 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:05.812023    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:19:05.824626 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:05.824604    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:19:06.298264 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:06.298241    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mvldf"
Sep 09 03:19:57.342776 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.342741    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fbbdbb45d-2prsm"]
Sep 09 03:19:57.346253 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.346233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fbbdbb45d-2prsm"
Sep 09 03:19:57.349436 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.349402    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 03:19:57.349570 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.349461    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 03:19:57.349570 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.349461    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 03:19:57.349570 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.349464    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l5zrp\""
Sep 09 03:19:57.355777 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.355756    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fbbdbb45d-2prsm"]
Sep 09 03:19:57.371002 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.370975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d940a15e-5591-40b6-9049-9e81a6f69920-tls-cert\") pod \"authorino-5fbbdbb45d-2prsm\" (UID: \"d940a15e-5591-40b6-9049-9e81a6f69920\") " pod="kuadrant-system/authorino-5fbbdbb45d-2prsm"
Sep 09 03:19:57.371118 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.371017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjxz\" (UniqueName: \"kubernetes.io/projected/d940a15e-5591-40b6-9049-9e81a6f69920-kube-api-access-fmjxz\") pod \"authorino-5fbbdbb45d-2prsm\" (UID: \"d940a15e-5591-40b6-9049-9e81a6f69920\") " pod="kuadrant-system/authorino-5fbbdbb45d-2prsm"
Sep 09 03:19:57.471824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.471791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d940a15e-5591-40b6-9049-9e81a6f69920-tls-cert\") pod \"authorino-5fbbdbb45d-2prsm\" (UID: \"d940a15e-5591-40b6-9049-9e81a6f69920\") " pod="kuadrant-system/authorino-5fbbdbb45d-2prsm"
Sep 09 03:19:57.471824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.471853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjxz\" (UniqueName: \"kubernetes.io/projected/d940a15e-5591-40b6-9049-9e81a6f69920-kube-api-access-fmjxz\") pod \"authorino-5fbbdbb45d-2prsm\" (UID: \"d940a15e-5591-40b6-9049-9e81a6f69920\") " pod="kuadrant-system/authorino-5fbbdbb45d-2prsm"
Sep 09 03:19:57.474404 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.474373    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d940a15e-5591-40b6-9049-9e81a6f69920-tls-cert\") pod \"authorino-5fbbdbb45d-2prsm\" (UID: \"d940a15e-5591-40b6-9049-9e81a6f69920\") " pod="kuadrant-system/authorino-5fbbdbb45d-2prsm"
Sep 09 03:19:57.479640 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.479611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjxz\" (UniqueName: \"kubernetes.io/projected/d940a15e-5591-40b6-9049-9e81a6f69920-kube-api-access-fmjxz\") pod \"authorino-5fbbdbb45d-2prsm\" (UID: \"d940a15e-5591-40b6-9049-9e81a6f69920\") " pod="kuadrant-system/authorino-5fbbdbb45d-2prsm"
Sep 09 03:19:57.655842 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.655804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fbbdbb45d-2prsm"
Sep 09 03:19:57.927917 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:57.927886    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fbbdbb45d-2prsm"]
Sep 09 03:19:57.930306 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:19:57.930271    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd940a15e_5591_40b6_9049_9e81a6f69920.slice/crio-686786ed233bfbe6aa5acc4c5b12af03422218a9aa88d00c8d9686722a071017 WatchSource:0}: Error finding container 686786ed233bfbe6aa5acc4c5b12af03422218a9aa88d00c8d9686722a071017: Status 404 returned error can't find the container with id 686786ed233bfbe6aa5acc4c5b12af03422218a9aa88d00c8d9686722a071017
Sep 09 03:19:58.540615 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:58.540581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fbbdbb45d-2prsm" event={"ID":"d940a15e-5591-40b6-9049-9e81a6f69920","Type":"ContainerStarted","Data":"686786ed233bfbe6aa5acc4c5b12af03422218a9aa88d00c8d9686722a071017"}
Sep 09 03:19:59.547853 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:59.547802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fbbdbb45d-2prsm" event={"ID":"d940a15e-5591-40b6-9049-9e81a6f69920","Type":"ContainerStarted","Data":"6a002bc7b4a98cca026805763a5b0b0f4af15964631b5ebb4915a8502a931364"}
Sep 09 03:19:59.565120 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:19:59.565055    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fbbdbb45d-2prsm" podStartSLOduration=1.87350079 podStartE2EDuration="2.56503349s" totalImagesPullingTime="691.5327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:19:59.561542343 +0000 UTC m=+849.075624495" watchObservedRunningTime="2026-09-09 03:19:59.56503349 +0000 UTC m=+849.079115640"
Sep 09 03:20:25.135623 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.135586    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"]
Sep 09 03:20:25.139086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.139067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.141417 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.141394    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-be414f-openshift-default-dockercfg-jxv7s\""
Sep 09 03:20:25.152279 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.152253    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"]
Sep 09 03:20:25.237080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.237032    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-credential-socket\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.237080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.237079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3b910-8cbc-4987-9935-adddbcc92826-istiod-ca-cert\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.237283 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.237168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-certs\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.237283 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.237210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3b910-8cbc-4987-9935-adddbcc92826-istio-podinfo\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.237569 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.237251    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhrd\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-kube-api-access-nbhrd\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.237908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.237884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-socket\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.238000 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.237986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-data\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.238069 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.238049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-envoy\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.238266 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.238243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-istio-token\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-envoy\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-istio-token\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-credential-socket\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3b910-8cbc-4987-9935-adddbcc92826-istiod-ca-cert\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-certs\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3b910-8cbc-4987-9935-adddbcc92826-istio-podinfo\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhrd\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-kube-api-access-nbhrd\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-socket\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339754 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-data\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339754 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-credential-socket\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339903 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-certs\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.339964 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.339919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-data\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.340032 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.340016    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-socket\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.340165 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.340144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3b910-8cbc-4987-9935-adddbcc92826-istiod-ca-cert\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.341655 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.341637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-envoy\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.341902 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.341879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3b910-8cbc-4987-9935-adddbcc92826-istio-podinfo\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.346664 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.346642    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-istio-token\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.346770 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.346670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhrd\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-kube-api-access-nbhrd\") pod \"e2e-autores-be414f-openshift-default-7986bc8586-hqvh6\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") " pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.450200 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.450121    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:25.734167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.734134    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"]
Sep 09 03:20:25.735315 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:20:25.735284    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b3b910_8cbc_4987_9935_adddbcc92826.slice/crio-1c5d8657f74db67243bb27f24a39a74ac78754f4014b9e39a45f437143153e03 WatchSource:0}: Error finding container 1c5d8657f74db67243bb27f24a39a74ac78754f4014b9e39a45f437143153e03: Status 404 returned error can't find the container with id 1c5d8657f74db67243bb27f24a39a74ac78754f4014b9e39a45f437143153e03
Sep 09 03:20:25.737417 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.737381    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:25.737508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.737446    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:25.737508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:25.737474    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:26.685104 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:26.685063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" event={"ID":"03b3b910-8cbc-4987-9935-adddbcc92826","Type":"ContainerStarted","Data":"56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1"}
Sep 09 03:20:26.685491 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:26.685121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" event={"ID":"03b3b910-8cbc-4987-9935-adddbcc92826","Type":"ContainerStarted","Data":"1c5d8657f74db67243bb27f24a39a74ac78754f4014b9e39a45f437143153e03"}
Sep 09 03:20:26.704992 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:26.704936    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" podStartSLOduration=1.704918379 podStartE2EDuration="1.704918379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:26.703480995 +0000 UTC m=+876.217563162" watchObservedRunningTime="2026-09-09 03:20:26.704918379 +0000 UTC m=+876.219000529"
Sep 09 03:20:27.451101 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:27.451059    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:27.456288 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:27.456261    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:27.691617 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:27.691549    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:27.693256 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:27.693224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:45.692037 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.691996    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"]
Sep 09 03:20:45.695609 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.695586    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.698673 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.698649    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-70784b-openshift-default-dockercfg-kcdzv\""
Sep 09 03:20:45.710942 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.710913    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"]
Sep 09 03:20:45.736255 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20920c8d-f6ec-473f-936f-c63d27834653-istiod-ca-cert\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.736420 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-envoy\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.736420 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736372    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-socket\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.736420 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-credential-socket\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.736536 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-data\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.736536 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20920c8d-f6ec-473f-936f-c63d27834653-istio-podinfo\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.736536 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-certs\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.736536 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-istio-token\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.736718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.736590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccg9g\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-kube-api-access-ccg9g\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.837499 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20920c8d-f6ec-473f-936f-c63d27834653-istiod-ca-cert\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.837687 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-envoy\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.837687 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-socket\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.837687 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-credential-socket\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.837687 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-data\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.837687 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20920c8d-f6ec-473f-936f-c63d27834653-istio-podinfo\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.837687 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-certs\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.838046 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-istio-token\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.838046 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.837751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccg9g\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-kube-api-access-ccg9g\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.838369 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.838348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20920c8d-f6ec-473f-936f-c63d27834653-istiod-ca-cert\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.838445 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.838413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-data\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.838528 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.838508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-credential-socket\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.838707 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.838686    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-certs\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.838794 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.838779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-socket\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.839966 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.839940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-envoy\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.840690 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.840662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20920c8d-f6ec-473f-936f-c63d27834653-istio-podinfo\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.846070 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.846044    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-istio-token\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:45.846537 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:45.846513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccg9g\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-kube-api-access-ccg9g\") pod \"e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") " pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:46.010206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:46.010125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:46.298379 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:46.298354    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"]
Sep 09 03:20:46.300502 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:20:46.300461    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20920c8d_f6ec_473f_936f_c63d27834653.slice/crio-7c90be2490ab4dc85ec494e786f7ece503459a448207fc8c94dc5a75c9ed9cd6 WatchSource:0}: Error finding container 7c90be2490ab4dc85ec494e786f7ece503459a448207fc8c94dc5a75c9ed9cd6: Status 404 returned error can't find the container with id 7c90be2490ab4dc85ec494e786f7ece503459a448207fc8c94dc5a75c9ed9cd6
Sep 09 03:20:46.302915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:46.302867    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:46.303027 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:46.302945    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:46.303027 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:46.302977    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:46.788088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:46.788051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" event={"ID":"20920c8d-f6ec-473f-936f-c63d27834653","Type":"ContainerStarted","Data":"d343bd88d002b8e14a7c257c6352e92d871385337ea8988514ca05bb92ea1415"}
Sep 09 03:20:46.788573 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:46.788122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" event={"ID":"20920c8d-f6ec-473f-936f-c63d27834653","Type":"ContainerStarted","Data":"7c90be2490ab4dc85ec494e786f7ece503459a448207fc8c94dc5a75c9ed9cd6"}
Sep 09 03:20:46.811630 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:46.811557    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podStartSLOduration=1.8115353870000002 podStartE2EDuration="1.811535387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:20:46.809396136 +0000 UTC m=+896.323478308" watchObservedRunningTime="2026-09-09 03:20:46.811535387 +0000 UTC m=+896.325617558"
Sep 09 03:20:47.011184 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.011156    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:47.016878 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.016844    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 09 03:20:47.017044 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.016902    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 09 03:20:47.793969 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.793937    2583 generic.go:356] "Generic (PLEG): container finished" podID="03b3b910-8cbc-4987-9935-adddbcc92826" containerID="56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1" exitCode=0
Sep 09 03:20:47.794383 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.794007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" event={"ID":"03b3b910-8cbc-4987-9935-adddbcc92826","Type":"ContainerDied","Data":"56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1"}
Sep 09 03:20:47.794528 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.794510    2583 scope.go:117] "RemoveContainer" containerID="56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1"
Sep 09 03:20:47.795527 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.795501    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:47.795599 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.795587    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:47.795639 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:47.795620    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:20:48.016278 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:48.016249    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:48.800811 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:48.800773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" event={"ID":"03b3b910-8cbc-4987-9935-adddbcc92826","Type":"ContainerStarted","Data":"dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe"}
Sep 09 03:20:48.801221 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:48.800996    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:48.801946 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:48.801925    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:20:49.450583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:49.450544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:49.450774 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:49.450755    2583 patch_prober.go:28] interesting pod/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 09 03:20:49.450819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:49.450794    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 09 03:20:50.451408 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:50.451368    2583 patch_prober.go:28] interesting pod/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 09 03:20:50.451908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:50.451438    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 09 03:20:50.942354 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:50.942328    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:20:50.942524 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:50.942500    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:20:50.948470 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:50.948448    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:20:50.948644 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:50.948627    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:20:51.451180 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:51.451141    2583 patch_prober.go:28] interesting pod/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 09 03:20:51.451353 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:51.451204    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 09 03:20:52.451528 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:52.451483    2583 patch_prober.go:28] interesting pod/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 09 03:20:52.452069 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:52.451556    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 09 03:20:53.125133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.125099    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"]
Sep 09 03:20:53.127703 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.127686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"
Sep 09 03:20:53.130091 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.130063    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:20:53.130299 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.130278    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:20:53.135648 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.135511    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"]
Sep 09 03:20:53.223586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.223547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0f5543-ce71-496b-86c6-bdc57bfc618e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb\" (UID: \"fe0f5543-ce71-496b-86c6-bdc57bfc618e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"
Sep 09 03:20:53.325217 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.325177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0f5543-ce71-496b-86c6-bdc57bfc618e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb\" (UID: \"fe0f5543-ce71-496b-86c6-bdc57bfc618e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"
Sep 09 03:20:53.325871 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.325854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0f5543-ce71-496b-86c6-bdc57bfc618e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb\" (UID: \"fe0f5543-ce71-496b-86c6-bdc57bfc618e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"
Sep 09 03:20:53.440057 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.439966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"
Sep 09 03:20:53.451502 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.451471    2583 patch_prober.go:28] interesting pod/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 09 03:20:53.451615 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.451537    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 09 03:20:53.734948 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.734920    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"]
Sep 09 03:20:53.736943 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:20:53.736914    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0f5543_ce71_496b_86c6_bdc57bfc618e.slice/crio-ced3fe1090bc0e7a2e1b3e8d235adc320d0014cab300913692be1292fcffd87b WatchSource:0}: Error finding container ced3fe1090bc0e7a2e1b3e8d235adc320d0014cab300913692be1292fcffd87b: Status 404 returned error can't find the container with id ced3fe1090bc0e7a2e1b3e8d235adc320d0014cab300913692be1292fcffd87b
Sep 09 03:20:53.828992 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:53.828951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb" event={"ID":"fe0f5543-ce71-496b-86c6-bdc57bfc618e","Type":"ContainerStarted","Data":"ced3fe1090bc0e7a2e1b3e8d235adc320d0014cab300913692be1292fcffd87b"}
Sep 09 03:20:54.557566 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:54.557525    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:54.558063 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:54.558042    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:54.559149 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:54.559124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:20:56.848919 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:56.848882    2583 generic.go:356] "Generic (PLEG): container finished" podID="fe0f5543-ce71-496b-86c6-bdc57bfc618e" containerID="7b82455e2e973a820c8d5cd07b0d08809c61afbd27339fcbabbfa219fd4760de" exitCode=0
Sep 09 03:20:56.849314 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:56.848981    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb" event={"ID":"fe0f5543-ce71-496b-86c6-bdc57bfc618e","Type":"ContainerDied","Data":"7b82455e2e973a820c8d5cd07b0d08809c61afbd27339fcbabbfa219fd4760de"}
Sep 09 03:20:58.020113 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:58.020087    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"
Sep 09 03:20:58.085854 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:58.085375    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0f5543-ce71-496b-86c6-bdc57bfc618e-maas-api-service-ca\") pod \"fe0f5543-ce71-496b-86c6-bdc57bfc618e\" (UID: \"fe0f5543-ce71-496b-86c6-bdc57bfc618e\") "
Sep 09 03:20:58.090299 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:58.086041    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0f5543-ce71-496b-86c6-bdc57bfc618e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe0f5543-ce71-496b-86c6-bdc57bfc618e" (UID: "fe0f5543-ce71-496b-86c6-bdc57bfc618e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:20:58.186454 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:58.186412    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0f5543-ce71-496b-86c6-bdc57bfc618e-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:20:58.860794 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:58.860759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb" event={"ID":"fe0f5543-ce71-496b-86c6-bdc57bfc618e","Type":"ContainerDied","Data":"ced3fe1090bc0e7a2e1b3e8d235adc320d0014cab300913692be1292fcffd87b"}
Sep 09 03:20:58.861010 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:58.860823    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced3fe1090bc0e7a2e1b3e8d235adc320d0014cab300913692be1292fcffd87b"
Sep 09 03:20:58.861010 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:20:58.860765    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"
Sep 09 03:21:00.521934 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:00.521905    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"]
Sep 09 03:21:00.525653 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:00.525626    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ecf63f7-ng8qb"]
Sep 09 03:21:01.008513 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.008475    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe0f5543-ce71-496b-86c6-bdc57bfc618e" path="/var/lib/kubelet/pods/fe0f5543-ce71-496b-86c6-bdc57bfc618e/volumes"
Sep 09 03:21:01.637492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.637463    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"]
Sep 09 03:21:01.637921 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.637906    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0f5543-ce71-496b-86c6-bdc57bfc618e" containerName="revoke-keys"
Sep 09 03:21:01.637921 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.637923    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0f5543-ce71-496b-86c6-bdc57bfc618e" containerName="revoke-keys"
Sep 09 03:21:01.638078 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.637985    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe0f5543-ce71-496b-86c6-bdc57bfc618e" containerName="revoke-keys"
Sep 09 03:21:01.640122 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.640104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.642715 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.642697    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0e6f9624-gw-openshift-default-dockercfg-kgpxf\""
Sep 09 03:21:01.653238 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.653216    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"]
Sep 09 03:21:01.723098 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pj2\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-kube-api-access-g5pj2\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.723245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-istio-token\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.723245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-socket\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.723245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-credential-socket\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.723245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-data\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.723394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84f0dd2-ed96-4572-9278-58facf137af5-istiod-ca-cert\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.723394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723342    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-envoy\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.723455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-certs\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.723455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.723428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84f0dd2-ed96-4572-9278-58facf137af5-istio-podinfo\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824477 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pj2\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-kube-api-access-g5pj2\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824652 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-istio-token\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824652 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-socket\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824652 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-credential-socket\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824652 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-data\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824652 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84f0dd2-ed96-4572-9278-58facf137af5-istiod-ca-cert\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824818 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-envoy\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824818 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-certs\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.824818 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.824743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84f0dd2-ed96-4572-9278-58facf137af5-istio-podinfo\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.825039 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.825011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-credential-socket\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.825134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.825109    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-socket\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.825284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.825241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-data\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.825428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.825334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-certs\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.825568 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.825548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84f0dd2-ed96-4572-9278-58facf137af5-istiod-ca-cert\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.827225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.827205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-envoy\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.827625 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.827609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84f0dd2-ed96-4572-9278-58facf137af5-istio-podinfo\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.835776 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.835753    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-istio-token\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.835776 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.835770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pj2\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-kube-api-access-g5pj2\") pod \"e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") " pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:01.950553 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:01.950469    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:02.259147 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:02.259122    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"]
Sep 09 03:21:02.261123 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:21:02.261081    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84f0dd2_ed96_4572_9278_58facf137af5.slice/crio-c6f9f891d8eb132216a2ef5d28588c6e16da3acd7a4bcca01620bead3eadb1f6 WatchSource:0}: Error finding container c6f9f891d8eb132216a2ef5d28588c6e16da3acd7a4bcca01620bead3eadb1f6: Status 404 returned error can't find the container with id c6f9f891d8eb132216a2ef5d28588c6e16da3acd7a4bcca01620bead3eadb1f6
Sep 09 03:21:02.263642 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:02.263608    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:02.263716 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:02.263693    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:02.263756 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:02.263722    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:02.892199 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:02.892165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" event={"ID":"f84f0dd2-ed96-4572-9278-58facf137af5","Type":"ContainerStarted","Data":"e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05"}
Sep 09 03:21:02.892569 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:02.892222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" event={"ID":"f84f0dd2-ed96-4572-9278-58facf137af5","Type":"ContainerStarted","Data":"c6f9f891d8eb132216a2ef5d28588c6e16da3acd7a4bcca01620bead3eadb1f6"}
Sep 09 03:21:02.917038 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:02.916987    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" podStartSLOduration=1.916969782 podStartE2EDuration="1.916969782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:02.915046479 +0000 UTC m=+912.429128676" watchObservedRunningTime="2026-09-09 03:21:02.916969782 +0000 UTC m=+912.431051931"
Sep 09 03:21:02.951498 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:02.951468    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:03.957560 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:03.957530    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:04.842047 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:04.842006    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 03:21:04.842222 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:04.842084    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 03:21:04.903573 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:04.903542    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:04.904507 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:04.904470    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:05.910136 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:05.910107    2583 generic.go:356] "Generic (PLEG): container finished" podID="20920c8d-f6ec-473f-936f-c63d27834653" containerID="d343bd88d002b8e14a7c257c6352e92d871385337ea8988514ca05bb92ea1415" exitCode=0
Sep 09 03:21:05.910562 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:05.910175    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" event={"ID":"20920c8d-f6ec-473f-936f-c63d27834653","Type":"ContainerDied","Data":"d343bd88d002b8e14a7c257c6352e92d871385337ea8988514ca05bb92ea1415"}
Sep 09 03:21:05.910682 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:05.910662    2583 scope.go:117] "RemoveContainer" containerID="d343bd88d002b8e14a7c257c6352e92d871385337ea8988514ca05bb92ea1415"
Sep 09 03:21:05.911945 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:05.911913    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:05.912090 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:05.912026    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:05.912090 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:05.912083    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:06.192682 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.192596    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"]
Sep 09 03:21:06.195937 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.195911    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.198515 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.198491    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-f99e31e6-kserve-self-signed-certs\""
Sep 09 03:21:06.206208 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.206180    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"]
Sep 09 03:21:06.272652 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.272613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-home\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.272884 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.272664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-dshm\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.272884 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.272733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tmp-dir\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.272884 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.272788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tls-certs\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.272884 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.272851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-model-cache\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.272884 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.272878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thg5m\" (UniqueName: \"kubernetes.io/projected/93b2ee50-ef13-4c73-8b87-9ce51863ef27-kube-api-access-thg5m\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.373710 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.373661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-model-cache\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.373926 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.373717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thg5m\" (UniqueName: \"kubernetes.io/projected/93b2ee50-ef13-4c73-8b87-9ce51863ef27-kube-api-access-thg5m\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.373926 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.373856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-home\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.373926 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.373889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-dshm\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.374096 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.373930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tmp-dir\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.374096 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.373955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tls-certs\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.374193 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.374145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-model-cache\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.374283 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.374249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-home\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.374432 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.374411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tmp-dir\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.376508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.376485    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-dshm\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.376805 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.376784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tls-certs\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.387064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.387043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thg5m\" (UniqueName: \"kubernetes.io/projected/93b2ee50-ef13-4c73-8b87-9ce51863ef27-kube-api-access-thg5m\") pod \"e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") " pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.507336 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.507164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:06.789812 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.789753    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"]
Sep 09 03:21:06.792808 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:21:06.792762    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93b2ee50_ef13_4c73_8b87_9ce51863ef27.slice/crio-8292d19c53a8728f6cc03c9b1151748a3ca05ac67ebf214309ff39c2e32c8558 WatchSource:0}: Error finding container 8292d19c53a8728f6cc03c9b1151748a3ca05ac67ebf214309ff39c2e32c8558: Status 404 returned error can't find the container with id 8292d19c53a8728f6cc03c9b1151748a3ca05ac67ebf214309ff39c2e32c8558
Sep 09 03:21:06.917438 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.917372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6" event={"ID":"93b2ee50-ef13-4c73-8b87-9ce51863ef27","Type":"ContainerStarted","Data":"67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e"}
Sep 09 03:21:06.917438 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.917414    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6" event={"ID":"93b2ee50-ef13-4c73-8b87-9ce51863ef27","Type":"ContainerStarted","Data":"8292d19c53a8728f6cc03c9b1151748a3ca05ac67ebf214309ff39c2e32c8558"}
Sep 09 03:21:06.919574 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.919548    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" event={"ID":"20920c8d-f6ec-473f-936f-c63d27834653","Type":"ContainerStarted","Data":"adb8e685fc8cd21ab1988cd7062f45f991701e42f67544253c44bea87143ec42"}
Sep 09 03:21:06.935150 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:06.935092    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6" podStartSLOduration=0.935076669 podStartE2EDuration="935.076669ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:06.933029751 +0000 UTC m=+916.447111912" watchObservedRunningTime="2026-09-09 03:21:06.935076669 +0000 UTC m=+916.449158814"
Sep 09 03:21:07.010780 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:07.010741    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:21:07.010975 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:07.010952    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 09 03:21:07.011046 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:07.010999    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 09 03:21:08.010666 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:08.010628    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 09 03:21:08.011148 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:08.010695    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 09 03:21:09.011006 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:09.010974    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 09 03:21:09.011466 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:09.011041    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 09 03:21:10.010544 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:10.010508    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 09 03:21:10.010734 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:10.010567    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 09 03:21:11.010858 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:11.010785    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 09 03:21:11.011320 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:11.010901    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 09 03:21:12.011231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:12.011192    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 09 03:21:12.011631 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:12.011249    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 09 03:21:13.014507 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:13.014478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:21:13.014927 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:13.014732    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:21:13.015483 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:13.015465    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:21:16.508499 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:16.508458    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:16.509021 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:16.508586    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:16.521529 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:16.521499    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:16.988323 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:16.988285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:21:26.027120 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:26.027038    2583 generic.go:356] "Generic (PLEG): container finished" podID="f84f0dd2-ed96-4572-9278-58facf137af5" containerID="e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05" exitCode=0
Sep 09 03:21:26.027512 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:26.027112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" event={"ID":"f84f0dd2-ed96-4572-9278-58facf137af5","Type":"ContainerDied","Data":"e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05"}
Sep 09 03:21:26.027554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:26.027521    2583 scope.go:117] "RemoveContainer" containerID="e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05"
Sep 09 03:21:26.028571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:26.028541    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:26.028671 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:26.028610    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:26.028671 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:26.028642    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:21:27.034001 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:27.033965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" event={"ID":"f84f0dd2-ed96-4572-9278-58facf137af5","Type":"ContainerStarted","Data":"9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106"}
Sep 09 03:21:27.950906 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:27.950869    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:27.951123 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:27.951105    2583 patch_prober.go:28] interesting pod/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:21:27.951167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:27.951146    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:21:28.951323 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:28.951281    2583 patch_prober.go:28] interesting pod/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:21:28.951722 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:28.951356    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:21:29.338671 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.338585    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"]
Sep 09 03:21:29.342007 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.341984    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"
Sep 09 03:21:29.344613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.344583    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:21:29.345441 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.345408    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:21:29.347572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.347548    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"]
Sep 09 03:21:29.417122 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.417068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effadb42-03d2-413c-a180-a260be2acc89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v\" (UID: \"effadb42-03d2-413c-a180-a260be2acc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"
Sep 09 03:21:29.518535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.518493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effadb42-03d2-413c-a180-a260be2acc89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v\" (UID: \"effadb42-03d2-413c-a180-a260be2acc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"
Sep 09 03:21:29.519198 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.519180    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effadb42-03d2-413c-a180-a260be2acc89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v\" (UID: \"effadb42-03d2-413c-a180-a260be2acc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"
Sep 09 03:21:29.655703 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.655667    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"
Sep 09 03:21:29.949161 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.949128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"]
Sep 09 03:21:29.950619 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:21:29.950586    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffadb42_03d2_413c_a180_a260be2acc89.slice/crio-faf963d8550187fc6fe6b733fb4a10351bdfdb184a872f0ad815e9c46eb7e369 WatchSource:0}: Error finding container faf963d8550187fc6fe6b733fb4a10351bdfdb184a872f0ad815e9c46eb7e369: Status 404 returned error can't find the container with id faf963d8550187fc6fe6b733fb4a10351bdfdb184a872f0ad815e9c46eb7e369
Sep 09 03:21:29.950950 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.950924    2583 patch_prober.go:28] interesting pod/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:21:29.951027 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:29.950982    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:21:30.049465 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:30.049435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v" event={"ID":"effadb42-03d2-413c-a180-a260be2acc89","Type":"ContainerStarted","Data":"faf963d8550187fc6fe6b733fb4a10351bdfdb184a872f0ad815e9c46eb7e369"}
Sep 09 03:21:30.951271 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:30.951230    2583 patch_prober.go:28] interesting pod/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:21:30.951441 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:30.951286    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:21:31.055800 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:31.055763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v" event={"ID":"effadb42-03d2-413c-a180-a260be2acc89","Type":"ContainerStarted","Data":"8298b4dd3a10cfecdf73cedc36550e6a41acc04bd9697b98eef8bb224b5b3d66"}
Sep 09 03:21:31.072989 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:31.072926    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v" podStartSLOduration=2.072896303 podStartE2EDuration="2.072896303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:31.070704806 +0000 UTC m=+940.584786976" watchObservedRunningTime="2026-09-09 03:21:31.072896303 +0000 UTC m=+940.586978456"
Sep 09 03:21:31.951590 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:31.951553    2583 patch_prober.go:28] interesting pod/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:21:31.951782 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:31.951609    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:21:32.064044 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:32.064010    2583 generic.go:356] "Generic (PLEG): container finished" podID="effadb42-03d2-413c-a180-a260be2acc89" containerID="8298b4dd3a10cfecdf73cedc36550e6a41acc04bd9697b98eef8bb224b5b3d66" exitCode=0
Sep 09 03:21:32.064438 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:32.064099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v" event={"ID":"effadb42-03d2-413c-a180-a260be2acc89","Type":"ContainerDied","Data":"8298b4dd3a10cfecdf73cedc36550e6a41acc04bd9697b98eef8bb224b5b3d66"}
Sep 09 03:21:32.954861 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:32.954814    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:32.955132 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:32.955110    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:32.955774 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:32.955757    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:33.209284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:33.209230    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"
Sep 09 03:21:33.257802 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:33.257769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effadb42-03d2-413c-a180-a260be2acc89-maas-api-service-ca\") pod \"effadb42-03d2-413c-a180-a260be2acc89\" (UID: \"effadb42-03d2-413c-a180-a260be2acc89\") "
Sep 09 03:21:33.258183 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:33.258160    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effadb42-03d2-413c-a180-a260be2acc89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "effadb42-03d2-413c-a180-a260be2acc89" (UID: "effadb42-03d2-413c-a180-a260be2acc89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:33.358656 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:33.358622    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effadb42-03d2-413c-a180-a260be2acc89-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:34.075604 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:34.075574    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"
Sep 09 03:21:34.075794 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:34.075620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v" event={"ID":"effadb42-03d2-413c-a180-a260be2acc89","Type":"ContainerDied","Data":"faf963d8550187fc6fe6b733fb4a10351bdfdb184a872f0ad815e9c46eb7e369"}
Sep 09 03:21:34.075794 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:34.075667    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faf963d8550187fc6fe6b733fb4a10351bdfdb184a872f0ad815e9c46eb7e369"
Sep 09 03:21:35.439590 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:35.439510    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"]
Sep 09 03:21:35.442128 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:35.442105    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0e6f9624-tbh4v"]
Sep 09 03:21:35.570520 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:35.570464    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"]
Sep 09 03:21:35.570799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:35.570772    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy" containerID="cri-o://9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106" gracePeriod=30
Sep 09 03:21:37.008311 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:37.008277    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="effadb42-03d2-413c-a180-a260be2acc89" path="/var/lib/kubelet/pods/effadb42-03d2-413c-a180-a260be2acc89/volumes"
Sep 09 03:21:40.927172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:40.927146    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:41.039072 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.038994    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-credential-socket\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039072 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039072    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-data\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039112    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-certs\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039140    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-istio-token\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039167    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84f0dd2-ed96-4572-9278-58facf137af5-istio-podinfo\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039205    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-envoy\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84f0dd2-ed96-4572-9278-58facf137af5-istiod-ca-cert\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039268    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5pj2\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-kube-api-access-g5pj2\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-socket\") pod \"f84f0dd2-ed96-4572-9278-58facf137af5\" (UID: \"f84f0dd2-ed96-4572-9278-58facf137af5\") "
Sep 09 03:21:41.039567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039311    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:41.039567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039339    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-data" (OuterVolumeSpecName: "istio-data") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:41.039567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039427    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:41.039740 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84f0dd2-ed96-4572-9278-58facf137af5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:21:41.039740 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039657    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.039740 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039676    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.039740 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:41.039740 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.039691    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.041701 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.041666    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:21:41.041813 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.041720    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-istio-token" (OuterVolumeSpecName: "istio-token") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:41.042005 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.041990    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-kube-api-access-g5pj2" (OuterVolumeSpecName: "kube-api-access-g5pj2") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "kube-api-access-g5pj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:21:41.042068 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.042043    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f84f0dd2-ed96-4572-9278-58facf137af5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f84f0dd2-ed96-4572-9278-58facf137af5" (UID: "f84f0dd2-ed96-4572-9278-58facf137af5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:21:41.114461 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.114429    2583 generic.go:356] "Generic (PLEG): container finished" podID="f84f0dd2-ed96-4572-9278-58facf137af5" containerID="9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106" exitCode=0
Sep 09 03:21:41.114660 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.114479    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"
Sep 09 03:21:41.114660 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.114531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" event={"ID":"f84f0dd2-ed96-4572-9278-58facf137af5","Type":"ContainerDied","Data":"9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106"}
Sep 09 03:21:41.114660 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.114593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk" event={"ID":"f84f0dd2-ed96-4572-9278-58facf137af5","Type":"ContainerDied","Data":"c6f9f891d8eb132216a2ef5d28588c6e16da3acd7a4bcca01620bead3eadb1f6"}
Sep 09 03:21:41.114660 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.114616    2583 scope.go:117] "RemoveContainer" containerID="9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106"
Sep 09 03:21:41.124747 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.124727    2583 scope.go:117] "RemoveContainer" containerID="e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05"
Sep 09 03:21:41.133271 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.133254    2583 scope.go:117] "RemoveContainer" containerID="9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106"
Sep 09 03:21:41.133542 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:21:41.133524    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106\": container with ID starting with 9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106 not found: ID does not exist" containerID="9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106"
Sep 09 03:21:41.133593 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.133551    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106"} err="failed to get container status \"9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106\": rpc error: code = NotFound desc = could not find container \"9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106\": container with ID starting with 9f7a9d1f5193ec3d3cbb15781c590cd70c7a8a23fba68d62b667ba919c482106 not found: ID does not exist"
Sep 09 03:21:41.133593 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.133568    2583 scope.go:117] "RemoveContainer" containerID="e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05"
Sep 09 03:21:41.133867 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:21:41.133851    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05\": container with ID starting with e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05 not found: ID does not exist" containerID="e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05"
Sep 09 03:21:41.133945 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.133871    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05"} err="failed to get container status \"e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05\": rpc error: code = NotFound desc = could not find container \"e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05\": container with ID starting with e696455fdd56f8dd6967d95ce62893ccf120c51df19d9c8611f97b63a6f09c05 not found: ID does not exist"
Sep 09 03:21:41.139595 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.139572    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"]
Sep 09 03:21:41.141152 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.140846    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.141152 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.140873    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84f0dd2-ed96-4572-9278-58facf137af5-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.141152 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.140888    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.141152 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.140925    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84f0dd2-ed96-4572-9278-58facf137af5-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.141152 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.140949    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5pj2\" (UniqueName: \"kubernetes.io/projected/f84f0dd2-ed96-4572-9278-58facf137af5-kube-api-access-g5pj2\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.141152 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.140964    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84f0dd2-ed96-4572-9278-58facf137af5-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:21:41.143270 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:41.143252    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0e6f9624-gw-openshift-default-5b97bb4bcb-6wbqk"]
Sep 09 03:21:43.008147 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:43.008112    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" path="/var/lib/kubelet/pods/f84f0dd2-ed96-4572-9278-58facf137af5/volumes"
Sep 09 03:21:52.187566 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.187528    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"]
Sep 09 03:21:52.188071 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.188033    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy"
Sep 09 03:21:52.188071 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.188048    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy"
Sep 09 03:21:52.188071 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.188058    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effadb42-03d2-413c-a180-a260be2acc89" containerName="revoke-keys"
Sep 09 03:21:52.188071 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.188064    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="effadb42-03d2-413c-a180-a260be2acc89" containerName="revoke-keys"
Sep 09 03:21:52.188206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.188144    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy"
Sep 09 03:21:52.188206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.188154    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy"
Sep 09 03:21:52.188206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.188161    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="effadb42-03d2-413c-a180-a260be2acc89" containerName="revoke-keys"
Sep 09 03:21:52.190997 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.190977    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.193455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.193434    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-be414f\"/\"openshift-service-ca.crt\""
Sep 09 03:21:52.194510 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.194484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-be414f\"/\"kube-root-ca.crt\""
Sep 09 03:21:52.194611 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.194550    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-be414f\"/\"default-dockercfg-wcrbh\""
Sep 09 03:21:52.194950 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.194936    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-be414f\"/\"autores-model-be414f-kserve-self-signed-certs\""
Sep 09 03:21:52.200679 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.200656    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"]
Sep 09 03:21:52.360205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.360166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-dshm\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.360386 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.360227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-model-cache\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.360386 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.360285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqxk\" (UniqueName: \"kubernetes.io/projected/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-kube-api-access-tfqxk\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.360386 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.360359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-home\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.360523 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.360387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tmp-dir\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.360523 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.360485    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tls-certs\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461157 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tls-certs\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461157 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-dshm\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461157 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-model-cache\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqxk\" (UniqueName: \"kubernetes.io/projected/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-kube-api-access-tfqxk\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-home\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tmp-dir\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461611 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-home\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461672 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-model-cache\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.461672 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.461649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tmp-dir\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.463616 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.463590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-dshm\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.463788 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.463772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tls-certs\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.468930 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.468906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqxk\" (UniqueName: \"kubernetes.io/projected/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-kube-api-access-tfqxk\") pod \"autores-model-be414f-kserve-57d96d6cbc-c8zbb\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") " pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.502212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.502173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:21:52.776699 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:52.776673    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"]
Sep 09 03:21:52.778873 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:21:52.778840    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8367d0f2_8b09_485a_8d6e_6e0d63f8bb32.slice/crio-fd356a9ab651e3bfca42123358d428c73597b800a55aedc133eb1a74c5ce1489 WatchSource:0}: Error finding container fd356a9ab651e3bfca42123358d428c73597b800a55aedc133eb1a74c5ce1489: Status 404 returned error can't find the container with id fd356a9ab651e3bfca42123358d428c73597b800a55aedc133eb1a74c5ce1489
Sep 09 03:21:53.185856 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:53.185797    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb" event={"ID":"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32","Type":"ContainerStarted","Data":"7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca"}
Sep 09 03:21:53.186035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:53.185876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb" event={"ID":"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32","Type":"ContainerStarted","Data":"fd356a9ab651e3bfca42123358d428c73597b800a55aedc133eb1a74c5ce1489"}
Sep 09 03:21:53.206353 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:21:53.206303    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb" podStartSLOduration=1.206288638 podStartE2EDuration="1.206288638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:53.204356438 +0000 UTC m=+962.718438605" watchObservedRunningTime="2026-09-09 03:21:53.206288638 +0000 UTC m=+962.720370786"
Sep 09 03:22:02.502883 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:02.502812    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:22:02.503485 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:02.502893    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:22:02.515562 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:02.515538    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:22:03.254977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:03.254938    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:22:04.815430 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.815388    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"]
Sep 09 03:22:04.816081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.816061    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy"
Sep 09 03:22:04.816148 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.816085    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84f0dd2-ed96-4572-9278-58facf137af5" containerName="istio-proxy"
Sep 09 03:22:04.818926 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.818906    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"
Sep 09 03:22:04.821333 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.821309    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:22:04.821450 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.821351    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:22:04.826160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.826137    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"]
Sep 09 03:22:04.888743 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.888709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0faa94-088b-4ca6-811b-eb0327678fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-189a872e-55c7l\" (UID: \"dc0faa94-088b-4ca6-811b-eb0327678fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"
Sep 09 03:22:04.989657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.989618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0faa94-088b-4ca6-811b-eb0327678fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-189a872e-55c7l\" (UID: \"dc0faa94-088b-4ca6-811b-eb0327678fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"
Sep 09 03:22:04.990263 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:04.990245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0faa94-088b-4ca6-811b-eb0327678fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-189a872e-55c7l\" (UID: \"dc0faa94-088b-4ca6-811b-eb0327678fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"
Sep 09 03:22:05.131748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:05.131716    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"
Sep 09 03:22:05.406165 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:05.406132    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"]
Sep 09 03:22:05.407713 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:22:05.407690    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0faa94_088b_4ca6_811b_eb0327678fcc.slice/crio-52ca9fc6716a97676c92ca2a4f39c8513891ffb93630f5d91f09aefc6605079e WatchSource:0}: Error finding container 52ca9fc6716a97676c92ca2a4f39c8513891ffb93630f5d91f09aefc6605079e: Status 404 returned error can't find the container with id 52ca9fc6716a97676c92ca2a4f39c8513891ffb93630f5d91f09aefc6605079e
Sep 09 03:22:06.258639 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:06.258604    2583 generic.go:356] "Generic (PLEG): container finished" podID="dc0faa94-088b-4ca6-811b-eb0327678fcc" containerID="471e8045b3b086bb7c431701dd4e248ae61e993c032c680a7eef2183ad0424d5" exitCode=0
Sep 09 03:22:06.259035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:06.258692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l" event={"ID":"dc0faa94-088b-4ca6-811b-eb0327678fcc","Type":"ContainerDied","Data":"471e8045b3b086bb7c431701dd4e248ae61e993c032c680a7eef2183ad0424d5"}
Sep 09 03:22:06.259035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:06.258747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l" event={"ID":"dc0faa94-088b-4ca6-811b-eb0327678fcc","Type":"ContainerStarted","Data":"52ca9fc6716a97676c92ca2a4f39c8513891ffb93630f5d91f09aefc6605079e"}
Sep 09 03:22:07.394259 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:07.394232    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"
Sep 09 03:22:07.516428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:07.516346    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0faa94-088b-4ca6-811b-eb0327678fcc-maas-api-service-ca\") pod \"dc0faa94-088b-4ca6-811b-eb0327678fcc\" (UID: \"dc0faa94-088b-4ca6-811b-eb0327678fcc\") "
Sep 09 03:22:07.516719 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:07.516697    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0faa94-088b-4ca6-811b-eb0327678fcc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc0faa94-088b-4ca6-811b-eb0327678fcc" (UID: "dc0faa94-088b-4ca6-811b-eb0327678fcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:07.617939 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:07.617899    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0faa94-088b-4ca6-811b-eb0327678fcc-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:22:08.270385 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:08.270357    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"
Sep 09 03:22:08.270620 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:08.270392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l" event={"ID":"dc0faa94-088b-4ca6-811b-eb0327678fcc","Type":"ContainerDied","Data":"52ca9fc6716a97676c92ca2a4f39c8513891ffb93630f5d91f09aefc6605079e"}
Sep 09 03:22:08.270620 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:08.270441    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ca9fc6716a97676c92ca2a4f39c8513891ffb93630f5d91f09aefc6605079e"
Sep 09 03:22:11.278675 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:11.278637    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"]
Sep 09 03:22:11.286425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:11.286386    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-189a872e-55c7l"]
Sep 09 03:22:13.008134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:13.008095    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc0faa94-088b-4ca6-811b-eb0327678fcc" path="/var/lib/kubelet/pods/dc0faa94-088b-4ca6-811b-eb0327678fcc/volumes"
Sep 09 03:22:26.732403 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.732356    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"]
Sep 09 03:22:26.733009 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.732959    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0faa94-088b-4ca6-811b-eb0327678fcc" containerName="revoke-keys"
Sep 09 03:22:26.733009 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.732983    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0faa94-088b-4ca6-811b-eb0327678fcc" containerName="revoke-keys"
Sep 09 03:22:26.733145 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.733107    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0faa94-088b-4ca6-811b-eb0327678fcc" containerName="revoke-keys"
Sep 09 03:22:26.735343 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.735323    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"
Sep 09 03:22:26.738028 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.738004    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:22:26.738157 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.738050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:22:26.745211 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.745186    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"]
Sep 09 03:22:26.804182 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.804141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-be414f-x6j72\" (UID: \"ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"
Sep 09 03:22:26.905543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.905506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-be414f-x6j72\" (UID: \"ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"
Sep 09 03:22:26.906178 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:26.906157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-be414f-x6j72\" (UID: \"ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"
Sep 09 03:22:27.048010 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:27.047899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"
Sep 09 03:22:27.319512 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:27.319485    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"]
Sep 09 03:22:27.322095 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:22:27.322067    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4a2ba8_dc3f_487d_a9e8_9adaaa10a809.slice/crio-691503859bd737a3b02eb988820a2728fd9ac565b9bbd94af1456c52d842875a WatchSource:0}: Error finding container 691503859bd737a3b02eb988820a2728fd9ac565b9bbd94af1456c52d842875a: Status 404 returned error can't find the container with id 691503859bd737a3b02eb988820a2728fd9ac565b9bbd94af1456c52d842875a
Sep 09 03:22:27.382351 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:27.382304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72" event={"ID":"ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809","Type":"ContainerStarted","Data":"691503859bd737a3b02eb988820a2728fd9ac565b9bbd94af1456c52d842875a"}
Sep 09 03:22:28.389846 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:28.389799    2583 generic.go:356] "Generic (PLEG): container finished" podID="ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809" containerID="be719551669e516f82331c8a2cd699707d3b5c2c9c26593845083a00370fcd64" exitCode=0
Sep 09 03:22:28.390217 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:28.389899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72" event={"ID":"ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809","Type":"ContainerDied","Data":"be719551669e516f82331c8a2cd699707d3b5c2c9c26593845083a00370fcd64"}
Sep 09 03:22:29.532588 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:29.532565    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"
Sep 09 03:22:29.635885 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:29.635814    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809-maas-api-service-ca\") pod \"ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809\" (UID: \"ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809\") "
Sep 09 03:22:29.636230 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:29.636200    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809" (UID: "ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:22:29.737413 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:29.737320    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:22:30.402600 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:30.402573    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"
Sep 09 03:22:30.402785 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:30.402610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72" event={"ID":"ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809","Type":"ContainerDied","Data":"691503859bd737a3b02eb988820a2728fd9ac565b9bbd94af1456c52d842875a"}
Sep 09 03:22:30.402785 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:30.402660    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="691503859bd737a3b02eb988820a2728fd9ac565b9bbd94af1456c52d842875a"
Sep 09 03:22:39.353441 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:39.353404    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"]
Sep 09 03:22:39.361264 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:39.361233    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-be414f-x6j72"]
Sep 09 03:22:41.007642 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:41.007605    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809" path="/var/lib/kubelet/pods/ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809/volumes"
Sep 09 03:22:44.705672 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:44.705642    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"]
Sep 09 03:22:59.728984 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:59.728939    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb" podUID="8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" containerName="main" containerID="cri-o://7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca" gracePeriod=45
Sep 09 03:22:59.987162 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:22:59.987102    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:23:00.144217 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144185    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tmp-dir\") pod \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") "
Sep 09 03:23:00.144402 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144245    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tls-certs\") pod \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") "
Sep 09 03:23:00.144402 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144336    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfqxk\" (UniqueName: \"kubernetes.io/projected/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-kube-api-access-tfqxk\") pod \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") "
Sep 09 03:23:00.144402 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144387    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-dshm\") pod \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") "
Sep 09 03:23:00.144571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144417    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-model-cache\") pod \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") "
Sep 09 03:23:00.144571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-home\") pod \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\" (UID: \"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32\") "
Sep 09 03:23:00.144571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144501    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" (UID: "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:00.144778 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-home" (OuterVolumeSpecName: "home") pod "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" (UID: "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:00.144894 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144719    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-model-cache" (OuterVolumeSpecName: "model-cache") pod "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" (UID: "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:00.144894 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.144778    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:00.146734 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.146711    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-dshm" (OuterVolumeSpecName: "dshm") pod "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" (UID: "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:00.146818 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.146766    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-kube-api-access-tfqxk" (OuterVolumeSpecName: "kube-api-access-tfqxk") pod "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" (UID: "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32"). InnerVolumeSpecName "kube-api-access-tfqxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:23:00.146818 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.146776    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" (UID: "8367d0f2-8b09-485a-8d6e-6e0d63f8bb32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:23:00.245933 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.245804    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqxk\" (UniqueName: \"kubernetes.io/projected/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-kube-api-access-tfqxk\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:00.245933 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.245876    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:00.245933 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.245887    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:00.245933 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.245895    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:00.245933 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.245906    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:00.566771 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.566675    2583 generic.go:356] "Generic (PLEG): container finished" podID="8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" containerID="7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca" exitCode=0
Sep 09 03:23:00.566771 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.566713    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"
Sep 09 03:23:00.566771 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.566720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb" event={"ID":"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32","Type":"ContainerDied","Data":"7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca"}
Sep 09 03:23:00.566771 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.566751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb" event={"ID":"8367d0f2-8b09-485a-8d6e-6e0d63f8bb32","Type":"ContainerDied","Data":"fd356a9ab651e3bfca42123358d428c73597b800a55aedc133eb1a74c5ce1489"}
Sep 09 03:23:00.566771 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.566770    2583 scope.go:117] "RemoveContainer" containerID="7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca"
Sep 09 03:23:00.584446 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.584413    2583 scope.go:117] "RemoveContainer" containerID="7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca"
Sep 09 03:23:00.584966 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:23:00.584942    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca\": container with ID starting with 7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca not found: ID does not exist" containerID="7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca"
Sep 09 03:23:00.585068 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.584970    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca"} err="failed to get container status \"7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca\": rpc error: code = NotFound desc = could not find container \"7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca\": container with ID starting with 7e86f7d226f31e10944be000dabf88043b5aa1c85df85db40534ee78678036ca not found: ID does not exist"
Sep 09 03:23:00.606490 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.606451    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"]
Sep 09 03:23:00.609627 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:00.609599    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-be414f/autores-model-be414f-kserve-57d96d6cbc-c8zbb"]
Sep 09 03:23:01.007779 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:01.007747    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" path="/var/lib/kubelet/pods/8367d0f2-8b09-485a-8d6e-6e0d63f8bb32/volumes"
Sep 09 03:23:15.924327 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:15.924291    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"]
Sep 09 03:23:21.629236 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:21.629197    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"]
Sep 09 03:23:21.629651 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:21.629441    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy" containerID="cri-o://dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe" gracePeriod=30
Sep 09 03:23:24.560106 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:24.560062    2583 patch_prober.go:28] interesting pod/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:23:24.560495 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:24.560128    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:23:26.886265 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:26.886239    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:23:27.007552 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007457    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-envoy\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.007552 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-certs\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.007552 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007525    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-data\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.007799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007556    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3b910-8cbc-4987-9935-adddbcc92826-istiod-ca-cert\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.007799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007584    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-istio-token\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.007799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007646    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbhrd\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-kube-api-access-nbhrd\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.007799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007680    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-credential-socket\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.007799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3b910-8cbc-4987-9935-adddbcc92826-istio-podinfo\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.007799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.007756    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-socket\") pod \"03b3b910-8cbc-4987-9935-adddbcc92826\" (UID: \"03b3b910-8cbc-4987-9935-adddbcc92826\") "
Sep 09 03:23:27.008155 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.008139    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:27.009172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.008331    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-data" (OuterVolumeSpecName: "istio-data") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:27.009172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.008537    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:27.009172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.008592    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:27.009172 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.009140    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b3b910-8cbc-4987-9935-adddbcc92826-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:23:27.010769 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.010684    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-kube-api-access-nbhrd" (OuterVolumeSpecName: "kube-api-access-nbhrd") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "kube-api-access-nbhrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:23:27.010769 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.010712    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-istio-token" (OuterVolumeSpecName: "istio-token") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:23:27.010769 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.010757    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:27.010981 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.010816    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03b3b910-8cbc-4987-9935-adddbcc92826-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03b3b910-8cbc-4987-9935-adddbcc92826" (UID: "03b3b910-8cbc-4987-9935-adddbcc92826"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:23:27.109309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109267    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.109309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109301    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.109309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109313    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.109563 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109324    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b3b910-8cbc-4987-9935-adddbcc92826-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.109563 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109336    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.109563 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109348    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhrd\" (UniqueName: \"kubernetes.io/projected/03b3b910-8cbc-4987-9935-adddbcc92826-kube-api-access-nbhrd\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.109563 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109359    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.109563 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109370    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b3b910-8cbc-4987-9935-adddbcc92826-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.109563 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.109380    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b3b910-8cbc-4987-9935-adddbcc92826-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:27.715265 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.715228    2583 generic.go:356] "Generic (PLEG): container finished" podID="03b3b910-8cbc-4987-9935-adddbcc92826" containerID="dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe" exitCode=0
Sep 09 03:23:27.715449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.715282    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"
Sep 09 03:23:27.715449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.715320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" event={"ID":"03b3b910-8cbc-4987-9935-adddbcc92826","Type":"ContainerDied","Data":"dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe"}
Sep 09 03:23:27.715449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.715378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6" event={"ID":"03b3b910-8cbc-4987-9935-adddbcc92826","Type":"ContainerDied","Data":"1c5d8657f74db67243bb27f24a39a74ac78754f4014b9e39a45f437143153e03"}
Sep 09 03:23:27.715449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.715402    2583 scope.go:117] "RemoveContainer" containerID="dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe"
Sep 09 03:23:27.724800 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.724784    2583 scope.go:117] "RemoveContainer" containerID="56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1"
Sep 09 03:23:27.733609 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.733589    2583 scope.go:117] "RemoveContainer" containerID="dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe"
Sep 09 03:23:27.733951 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:23:27.733922    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe\": container with ID starting with dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe not found: ID does not exist" containerID="dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe"
Sep 09 03:23:27.734050 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.733957    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe"} err="failed to get container status \"dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe\": rpc error: code = NotFound desc = could not find container \"dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe\": container with ID starting with dfdcab2871a8f18a6ce548b29f861c9b1dbff858b244cbf4547199981dc15dbe not found: ID does not exist"
Sep 09 03:23:27.734050 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.733975    2583 scope.go:117] "RemoveContainer" containerID="56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1"
Sep 09 03:23:27.734238 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:23:27.734224    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1\": container with ID starting with 56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1 not found: ID does not exist" containerID="56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1"
Sep 09 03:23:27.734278 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.734243    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1"} err="failed to get container status \"56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1\": rpc error: code = NotFound desc = could not find container \"56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1\": container with ID starting with 56fde260c9174a6d421fafdfe3b4ef202dba00305cc35b10c706011b973034c1 not found: ID does not exist"
Sep 09 03:23:27.738623 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.738599    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"]
Sep 09 03:23:27.745114 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:27.745088    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-be414f-openshift-default-7986bc8586-hqvh6"]
Sep 09 03:23:29.008414 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:29.008378    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" path="/var/lib/kubelet/pods/03b3b910-8cbc-4987-9935-adddbcc92826/volumes"
Sep 09 03:23:30.946305 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:30.946245    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6" podUID="93b2ee50-ef13-4c73-8b87-9ce51863ef27" containerName="main" containerID="cri-o://67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e" gracePeriod=45
Sep 09 03:23:31.199323 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.199246    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:23:31.354162 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354126    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-dshm\") pod \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") "
Sep 09 03:23:31.354325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354216    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-model-cache\") pod \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") "
Sep 09 03:23:31.354325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354253    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-home\") pod \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") "
Sep 09 03:23:31.354325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354285    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thg5m\" (UniqueName: \"kubernetes.io/projected/93b2ee50-ef13-4c73-8b87-9ce51863ef27-kube-api-access-thg5m\") pod \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") "
Sep 09 03:23:31.354436 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354366    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tls-certs\") pod \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") "
Sep 09 03:23:31.354436 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354415    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tmp-dir\") pod \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\" (UID: \"93b2ee50-ef13-4c73-8b87-9ce51863ef27\") "
Sep 09 03:23:31.354531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-model-cache" (OuterVolumeSpecName: "model-cache") pod "93b2ee50-ef13-4c73-8b87-9ce51863ef27" (UID: "93b2ee50-ef13-4c73-8b87-9ce51863ef27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:31.354586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354517    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-home" (OuterVolumeSpecName: "home") pod "93b2ee50-ef13-4c73-8b87-9ce51863ef27" (UID: "93b2ee50-ef13-4c73-8b87-9ce51863ef27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:31.354739 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354715    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93b2ee50-ef13-4c73-8b87-9ce51863ef27" (UID: "93b2ee50-ef13-4c73-8b87-9ce51863ef27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:31.354911 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354809    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:31.354911 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354843    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:31.354911 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.354856    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:31.356624 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.356600    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-dshm" (OuterVolumeSpecName: "dshm") pod "93b2ee50-ef13-4c73-8b87-9ce51863ef27" (UID: "93b2ee50-ef13-4c73-8b87-9ce51863ef27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:23:31.356760 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.356644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93b2ee50-ef13-4c73-8b87-9ce51863ef27" (UID: "93b2ee50-ef13-4c73-8b87-9ce51863ef27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:23:31.356760 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.356743    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b2ee50-ef13-4c73-8b87-9ce51863ef27-kube-api-access-thg5m" (OuterVolumeSpecName: "kube-api-access-thg5m") pod "93b2ee50-ef13-4c73-8b87-9ce51863ef27" (UID: "93b2ee50-ef13-4c73-8b87-9ce51863ef27"). InnerVolumeSpecName "kube-api-access-thg5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:23:31.456281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.456184    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thg5m\" (UniqueName: \"kubernetes.io/projected/93b2ee50-ef13-4c73-8b87-9ce51863ef27-kube-api-access-thg5m\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:31.456281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.456221    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b2ee50-ef13-4c73-8b87-9ce51863ef27-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:31.456281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.456231    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b2ee50-ef13-4c73-8b87-9ce51863ef27-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:23:31.739029 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.738945    2583 generic.go:356] "Generic (PLEG): container finished" podID="93b2ee50-ef13-4c73-8b87-9ce51863ef27" containerID="67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e" exitCode=0
Sep 09 03:23:31.739029 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.738992    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"
Sep 09 03:23:31.739214 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.739033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6" event={"ID":"93b2ee50-ef13-4c73-8b87-9ce51863ef27","Type":"ContainerDied","Data":"67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e"}
Sep 09 03:23:31.739214 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.739087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6" event={"ID":"93b2ee50-ef13-4c73-8b87-9ce51863ef27","Type":"ContainerDied","Data":"8292d19c53a8728f6cc03c9b1151748a3ca05ac67ebf214309ff39c2e32c8558"}
Sep 09 03:23:31.739214 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.739110    2583 scope.go:117] "RemoveContainer" containerID="67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e"
Sep 09 03:23:31.748698 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.748678    2583 scope.go:117] "RemoveContainer" containerID="67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e"
Sep 09 03:23:31.749040 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:23:31.749021    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e\": container with ID starting with 67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e not found: ID does not exist" containerID="67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e"
Sep 09 03:23:31.749100 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.749049    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e"} err="failed to get container status \"67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e\": rpc error: code = NotFound desc = could not find container \"67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e\": container with ID starting with 67ff0cbf57240930157a04662871ecf897fa171c1da0558f17dafc4550cb459e not found: ID does not exist"
Sep 09 03:23:31.760272 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.760244    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"]
Sep 09 03:23:31.767339 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:31.767312    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-f99e31e6-kserve-7b8bcc9645-4dvq6"]
Sep 09 03:23:32.589112 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589076    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"]
Sep 09 03:23:32.589531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589464    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy"
Sep 09 03:23:32.589531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589479    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy"
Sep 09 03:23:32.589531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589492    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b2ee50-ef13-4c73-8b87-9ce51863ef27" containerName="main"
Sep 09 03:23:32.589531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589500    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b2ee50-ef13-4c73-8b87-9ce51863ef27" containerName="main"
Sep 09 03:23:32.589531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589509    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809" containerName="revoke-keys"
Sep 09 03:23:32.589531 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589517    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809" containerName="revoke-keys"
Sep 09 03:23:32.589748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589536    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" containerName="main"
Sep 09 03:23:32.589748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589544    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" containerName="main"
Sep 09 03:23:32.589748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589640    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy"
Sep 09 03:23:32.589748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589649    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b2ee50-ef13-4c73-8b87-9ce51863ef27" containerName="main"
Sep 09 03:23:32.589748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589656    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8367d0f2-8b09-485a-8d6e-6e0d63f8bb32" containerName="main"
Sep 09 03:23:32.589748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.589663    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4a2ba8-dc3f-487d-a9e8-9adaaa10a809" containerName="revoke-keys"
Sep 09 03:23:32.593776 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.593756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.597117 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.597095    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-70784b\"/\"openshift-service-ca.crt\""
Sep 09 03:23:32.597257 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.597134    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-70784b\"/\"default-dockercfg-xmng6\""
Sep 09 03:23:32.597257 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.597149    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-70784b\"/\"test-model-70784b-kserve-self-signed-certs\""
Sep 09 03:23:32.597351 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.597265    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-70784b\"/\"kube-root-ca.crt\""
Sep 09 03:23:32.601359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.601336    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"]
Sep 09 03:23:32.668657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.668613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-tmp-dir\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.668869 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.668678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-dshm\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.668869 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.668716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7d064c-e30e-483c-929a-248f0d43bdf4-tls-certs\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.668869 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.668794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5tx\" (UniqueName: \"kubernetes.io/projected/2d7d064c-e30e-483c-929a-248f0d43bdf4-kube-api-access-hs5tx\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.668869 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.668860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-model-cache\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.669026 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.668882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-home\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.769762 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.769714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-dshm\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.769986 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.769773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7d064c-e30e-483c-929a-248f0d43bdf4-tls-certs\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.769986 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.769862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5tx\" (UniqueName: \"kubernetes.io/projected/2d7d064c-e30e-483c-929a-248f0d43bdf4-kube-api-access-hs5tx\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.769986 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.769892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-model-cache\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.769986 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.769915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-home\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.770199 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.770006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-tmp-dir\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.770323 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.770295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-model-cache\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.770372 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.770329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-tmp-dir\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.770372 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.770366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-home\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.772355 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.772332    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-dshm\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.772545 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.772521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7d064c-e30e-483c-929a-248f0d43bdf4-tls-certs\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.781104 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.781080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5tx\" (UniqueName: \"kubernetes.io/projected/2d7d064c-e30e-483c-929a-248f0d43bdf4-kube-api-access-hs5tx\") pod \"test-model-70784b-kserve-6ff8975589-ltx2n\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") " pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:32.904478 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:32.904433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:33.007783 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:33.007751    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b2ee50-ef13-4c73-8b87-9ce51863ef27" path="/var/lib/kubelet/pods/93b2ee50-ef13-4c73-8b87-9ce51863ef27/volumes"
Sep 09 03:23:33.179166 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:33.179138    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"]
Sep 09 03:23:33.181181 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:23:33.181153    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d7d064c_e30e_483c_929a_248f0d43bdf4.slice/crio-914ad2ef89303f4cb7cbd82675a0cd3e9f7620c408b29f4cbb5cf33ee3c45f18 WatchSource:0}: Error finding container 914ad2ef89303f4cb7cbd82675a0cd3e9f7620c408b29f4cbb5cf33ee3c45f18: Status 404 returned error can't find the container with id 914ad2ef89303f4cb7cbd82675a0cd3e9f7620c408b29f4cbb5cf33ee3c45f18
Sep 09 03:23:33.761148 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:33.761102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n" event={"ID":"2d7d064c-e30e-483c-929a-248f0d43bdf4","Type":"ContainerStarted","Data":"e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702"}
Sep 09 03:23:33.761148 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:33.761156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n" event={"ID":"2d7d064c-e30e-483c-929a-248f0d43bdf4","Type":"ContainerStarted","Data":"914ad2ef89303f4cb7cbd82675a0cd3e9f7620c408b29f4cbb5cf33ee3c45f18"}
Sep 09 03:23:33.780288 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:33.780216    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n" podStartSLOduration=1.7801991190000002 podStartE2EDuration="1.780199119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:23:33.779448164 +0000 UTC m=+1063.293530351" watchObservedRunningTime="2026-09-09 03:23:33.780199119 +0000 UTC m=+1063.294281269"
Sep 09 03:23:42.904659 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:42.904561    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:42.904659 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:42.904619    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:42.923021 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:42.922985    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:23:43.826677 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:23:43.826637    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:24:04.038597 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.038565    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"]
Sep 09 03:24:04.039003 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.038979    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy"
Sep 09 03:24:04.039003 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.038992    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy"
Sep 09 03:24:04.039095 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.039084    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b3b910-8cbc-4987-9935-adddbcc92826" containerName="istio-proxy"
Sep 09 03:24:04.043587 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.043570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.048406 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.048385    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4ce4c1\"/\"openshift-service-ca.crt\""
Sep 09 03:24:04.049679 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.049658    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4ce4c1\"/\"default-dockercfg-977gd\""
Sep 09 03:24:04.049809 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.049677    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4ce4c1\"/\"explicit-model-4ce4c1-kserve-self-signed-certs\""
Sep 09 03:24:04.049809 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.049702    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4ce4c1\"/\"kube-root-ca.crt\""
Sep 09 03:24:04.081711 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.081667    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"]
Sep 09 03:24:04.177759 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.177719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-model-cache\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.177993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.177780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxg9\" (UniqueName: \"kubernetes.io/projected/935983ac-efa8-46db-af2b-84ca67cfd785-kube-api-access-gvxg9\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.177993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.177880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-dshm\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.177993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.177951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/935983ac-efa8-46db-af2b-84ca67cfd785-tls-certs\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.178142 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.178057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-tmp-dir\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.178142 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.178132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-home\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.278726 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.278686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/935983ac-efa8-46db-af2b-84ca67cfd785-tls-certs\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.278726 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.278732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-tmp-dir\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.278988 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.278768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-home\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.278988 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.278811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-model-cache\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.278988 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.278861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxg9\" (UniqueName: \"kubernetes.io/projected/935983ac-efa8-46db-af2b-84ca67cfd785-kube-api-access-gvxg9\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.278988 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.278898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-dshm\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.279225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.279198    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-tmp-dir\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.279288 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.279255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-model-cache\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.279330 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.279304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-home\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.281293 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.281263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-dshm\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.281402 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.281393    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/935983ac-efa8-46db-af2b-84ca67cfd785-tls-certs\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.296126 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.296074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxg9\" (UniqueName: \"kubernetes.io/projected/935983ac-efa8-46db-af2b-84ca67cfd785-kube-api-access-gvxg9\") pod \"explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") " pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.352993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.352963    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:04.627957 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.627921    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"]
Sep 09 03:24:04.630576 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:24:04.630546    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod935983ac_efa8_46db_af2b_84ca67cfd785.slice/crio-46c2b70449b2a8cc9eb8a66f872a224fd0f581c03a52f4e50c5fc08992c4b96b WatchSource:0}: Error finding container 46c2b70449b2a8cc9eb8a66f872a224fd0f581c03a52f4e50c5fc08992c4b96b: Status 404 returned error can't find the container with id 46c2b70449b2a8cc9eb8a66f872a224fd0f581c03a52f4e50c5fc08992c4b96b
Sep 09 03:24:04.924972 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.924874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd" event={"ID":"935983ac-efa8-46db-af2b-84ca67cfd785","Type":"ContainerStarted","Data":"0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0"}
Sep 09 03:24:04.924972 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.924918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd" event={"ID":"935983ac-efa8-46db-af2b-84ca67cfd785","Type":"ContainerStarted","Data":"46c2b70449b2a8cc9eb8a66f872a224fd0f581c03a52f4e50c5fc08992c4b96b"}
Sep 09 03:24:04.966448 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:04.966396    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd" podStartSLOduration=1.966375829 podStartE2EDuration="1.966375829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:04.964439011 +0000 UTC m=+1094.478521207" watchObservedRunningTime="2026-09-09 03:24:04.966375829 +0000 UTC m=+1094.480457979"
Sep 09 03:24:14.353848 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:14.353792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:14.354313 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:14.353876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:14.366695 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:14.366665    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:14.990045 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:14.990012    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:26.647859 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.647809    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"]
Sep 09 03:24:26.766622 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.766590    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"]
Sep 09 03:24:26.769572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.769555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"
Sep 09 03:24:26.772128 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.772110    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:24:26.772225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.772150    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:24:26.777492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.777468    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"]
Sep 09 03:24:26.891748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.891711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5f4553-67c3-46b4-8279-1b702c06044c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d\" (UID: \"cc5f4553-67c3-46b4-8279-1b702c06044c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"
Sep 09 03:24:26.992653 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.992565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5f4553-67c3-46b4-8279-1b702c06044c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d\" (UID: \"cc5f4553-67c3-46b4-8279-1b702c06044c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"
Sep 09 03:24:26.993168 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:26.993151    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5f4553-67c3-46b4-8279-1b702c06044c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d\" (UID: \"cc5f4553-67c3-46b4-8279-1b702c06044c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"
Sep 09 03:24:27.082690 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:27.082656    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"
Sep 09 03:24:27.354710 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:27.354680    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"]
Sep 09 03:24:27.356865 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:24:27.356807    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5f4553_67c3_46b4_8279_1b702c06044c.slice/crio-d867462dd21421d6f256a99f1c4afbc846e1ee22b7755703c9fc9fbd82bb021b WatchSource:0}: Error finding container d867462dd21421d6f256a99f1c4afbc846e1ee22b7755703c9fc9fbd82bb021b: Status 404 returned error can't find the container with id d867462dd21421d6f256a99f1c4afbc846e1ee22b7755703c9fc9fbd82bb021b
Sep 09 03:24:28.049199 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:28.049161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d" event={"ID":"cc5f4553-67c3-46b4-8279-1b702c06044c","Type":"ContainerStarted","Data":"70905f74a0cb25fd495b3bea1c964f5a5a7a450f45fea64a5a6852ea91dfbc2a"}
Sep 09 03:24:28.049693 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:28.049220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d" event={"ID":"cc5f4553-67c3-46b4-8279-1b702c06044c","Type":"ContainerStarted","Data":"d867462dd21421d6f256a99f1c4afbc846e1ee22b7755703c9fc9fbd82bb021b"}
Sep 09 03:24:28.065963 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:28.065917    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d" podStartSLOduration=2.065901846 podStartE2EDuration="2.065901846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:28.064902436 +0000 UTC m=+1117.578984604" watchObservedRunningTime="2026-09-09 03:24:28.065901846 +0000 UTC m=+1117.579983996"
Sep 09 03:24:29.056529 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:29.056494    2583 generic.go:356] "Generic (PLEG): container finished" podID="cc5f4553-67c3-46b4-8279-1b702c06044c" containerID="70905f74a0cb25fd495b3bea1c964f5a5a7a450f45fea64a5a6852ea91dfbc2a" exitCode=0
Sep 09 03:24:29.056948 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:29.056578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d" event={"ID":"cc5f4553-67c3-46b4-8279-1b702c06044c","Type":"ContainerDied","Data":"70905f74a0cb25fd495b3bea1c964f5a5a7a450f45fea64a5a6852ea91dfbc2a"}
Sep 09 03:24:30.196452 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:30.196429    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"
Sep 09 03:24:30.225217 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:30.225191    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5f4553-67c3-46b4-8279-1b702c06044c-maas-api-service-ca\") pod \"cc5f4553-67c3-46b4-8279-1b702c06044c\" (UID: \"cc5f4553-67c3-46b4-8279-1b702c06044c\") "
Sep 09 03:24:30.225508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:30.225488    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5f4553-67c3-46b4-8279-1b702c06044c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc5f4553-67c3-46b4-8279-1b702c06044c" (UID: "cc5f4553-67c3-46b4-8279-1b702c06044c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:30.325890 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:30.325807    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5f4553-67c3-46b4-8279-1b702c06044c-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:24:31.068241 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:31.068214    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"
Sep 09 03:24:31.068411 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:31.068211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d" event={"ID":"cc5f4553-67c3-46b4-8279-1b702c06044c","Type":"ContainerDied","Data":"d867462dd21421d6f256a99f1c4afbc846e1ee22b7755703c9fc9fbd82bb021b"}
Sep 09 03:24:31.068411 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:31.068335    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d867462dd21421d6f256a99f1c4afbc846e1ee22b7755703c9fc9fbd82bb021b"
Sep 09 03:24:33.721571 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.721538    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"]
Sep 09 03:24:33.721988 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.721933    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5f4553-67c3-46b4-8279-1b702c06044c" containerName="revoke-keys"
Sep 09 03:24:33.721988 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.721945    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5f4553-67c3-46b4-8279-1b702c06044c" containerName="revoke-keys"
Sep 09 03:24:33.722062 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.722014    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5f4553-67c3-46b4-8279-1b702c06044c" containerName="revoke-keys"
Sep 09 03:24:33.725236 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.725220    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.728545 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.728509    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-57aecdff-openshift-default-dockercfg-9rfg5\""
Sep 09 03:24:33.759632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.759597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-socket\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.759795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.759634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvz2x\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-kube-api-access-jvz2x\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.759795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.759693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-data\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.759795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.759744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-envoy\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.759795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.759761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-token\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.759970 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.759909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-podinfo\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.760004 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.759974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-credential-socket\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.760004 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.759995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-certs\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.760068 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.760017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istiod-ca-cert\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.784326 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.784293    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"]
Sep 09 03:24:33.860685 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.860638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-socket\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.860685 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.860688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvz2x\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-kube-api-access-jvz2x\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861008 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.860735    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-data\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861008 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.860793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-envoy\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861008 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.860815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-token\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861008 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.860911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-podinfo\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861008 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.860951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-credential-socket\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861008 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.860976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-certs\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861008 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.861008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istiod-ca-cert\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861376 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.861094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-socket\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861376 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.861270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-data\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861376 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.861326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-credential-socket\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.861375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-certs\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.861711 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.861691    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istiod-ca-cert\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.863222 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.863202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-envoy\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.863619 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.863600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-podinfo\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.881697 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.881671    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-token\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:33.881809 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:33.881771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvz2x\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-kube-api-access-jvz2x\") pod \"e2e-mt-57aecdff-openshift-default-5f5859d496-89dct\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") " pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:34.036526 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:34.036436    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:34.318335 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:34.318303    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"]
Sep 09 03:24:34.320515 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:24:34.320487    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129c7d5b_784c_4ea0_99b6_b1ddfffb8449.slice/crio-2fbb34032d2991c94bb991825ee40a08e0d0f1f0767421b86a8da58e924b75b2 WatchSource:0}: Error finding container 2fbb34032d2991c94bb991825ee40a08e0d0f1f0767421b86a8da58e924b75b2: Status 404 returned error can't find the container with id 2fbb34032d2991c94bb991825ee40a08e0d0f1f0767421b86a8da58e924b75b2
Sep 09 03:24:34.322704 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:34.322668    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:24:34.322804 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:34.322748    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:24:34.322804 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:34.322796    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:24:35.091916 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:35.091869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" event={"ID":"129c7d5b-784c-4ea0-99b6-b1ddfffb8449","Type":"ContainerStarted","Data":"a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c"}
Sep 09 03:24:35.092309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:35.091940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" event={"ID":"129c7d5b-784c-4ea0-99b6-b1ddfffb8449","Type":"ContainerStarted","Data":"2fbb34032d2991c94bb991825ee40a08e0d0f1f0767421b86a8da58e924b75b2"}
Sep 09 03:24:35.115501 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:35.115453    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podStartSLOduration=2.115437867 podStartE2EDuration="2.115437867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:35.11354092 +0000 UTC m=+1124.627623082" watchObservedRunningTime="2026-09-09 03:24:35.115437867 +0000 UTC m=+1124.629520016"
Sep 09 03:24:36.036573 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:36.036536    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:36.041770 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:36.041746    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:36.097890 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:36.097859    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:36.099408 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:36.099386    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:37.495504 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:37.495470    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"]
Sep 09 03:24:37.498102 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:37.498075    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ce4c1-8x28d"]
Sep 09 03:24:39.014464 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:39.014419    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc5f4553-67c3-46b4-8279-1b702c06044c" path="/var/lib/kubelet/pods/cc5f4553-67c3-46b4-8279-1b702c06044c/volumes"
Sep 09 03:24:41.668558 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:41.668496    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd" podUID="935983ac-efa8-46db-af2b-84ca67cfd785" containerName="main" containerID="cri-o://0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0" gracePeriod=45
Sep 09 03:24:41.919047 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:41.918982    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:42.049377 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049343    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-dshm\") pod \"935983ac-efa8-46db-af2b-84ca67cfd785\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") "
Sep 09 03:24:42.049526 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-model-cache\") pod \"935983ac-efa8-46db-af2b-84ca67cfd785\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") "
Sep 09 03:24:42.049526 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049488    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/935983ac-efa8-46db-af2b-84ca67cfd785-tls-certs\") pod \"935983ac-efa8-46db-af2b-84ca67cfd785\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") "
Sep 09 03:24:42.049526 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-home\") pod \"935983ac-efa8-46db-af2b-84ca67cfd785\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") "
Sep 09 03:24:42.049645 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049550    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-tmp-dir\") pod \"935983ac-efa8-46db-af2b-84ca67cfd785\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") "
Sep 09 03:24:42.049645 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049606    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvxg9\" (UniqueName: \"kubernetes.io/projected/935983ac-efa8-46db-af2b-84ca67cfd785-kube-api-access-gvxg9\") pod \"935983ac-efa8-46db-af2b-84ca67cfd785\" (UID: \"935983ac-efa8-46db-af2b-84ca67cfd785\") "
Sep 09 03:24:42.049786 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049740    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-model-cache" (OuterVolumeSpecName: "model-cache") pod "935983ac-efa8-46db-af2b-84ca67cfd785" (UID: "935983ac-efa8-46db-af2b-84ca67cfd785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:42.049883 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049817    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-home" (OuterVolumeSpecName: "home") pod "935983ac-efa8-46db-af2b-84ca67cfd785" (UID: "935983ac-efa8-46db-af2b-84ca67cfd785"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:42.049952 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.049908    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "935983ac-efa8-46db-af2b-84ca67cfd785" (UID: "935983ac-efa8-46db-af2b-84ca67cfd785"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:42.050059 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.050038    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:24:42.050206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.050062    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:24:42.050206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.050076    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:24:42.052150 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.052117    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935983ac-efa8-46db-af2b-84ca67cfd785-kube-api-access-gvxg9" (OuterVolumeSpecName: "kube-api-access-gvxg9") pod "935983ac-efa8-46db-af2b-84ca67cfd785" (UID: "935983ac-efa8-46db-af2b-84ca67cfd785"). InnerVolumeSpecName "kube-api-access-gvxg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:24:42.052288 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.052266    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935983ac-efa8-46db-af2b-84ca67cfd785-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "935983ac-efa8-46db-af2b-84ca67cfd785" (UID: "935983ac-efa8-46db-af2b-84ca67cfd785"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:24:42.052346 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.052269    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-dshm" (OuterVolumeSpecName: "dshm") pod "935983ac-efa8-46db-af2b-84ca67cfd785" (UID: "935983ac-efa8-46db-af2b-84ca67cfd785"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:24:42.129802 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.129752    2583 generic.go:356] "Generic (PLEG): container finished" podID="935983ac-efa8-46db-af2b-84ca67cfd785" containerID="0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0" exitCode=0
Sep 09 03:24:42.130014 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.129808    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"
Sep 09 03:24:42.130014 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.129812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd" event={"ID":"935983ac-efa8-46db-af2b-84ca67cfd785","Type":"ContainerDied","Data":"0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0"}
Sep 09 03:24:42.130014 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.129883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd" event={"ID":"935983ac-efa8-46db-af2b-84ca67cfd785","Type":"ContainerDied","Data":"46c2b70449b2a8cc9eb8a66f872a224fd0f581c03a52f4e50c5fc08992c4b96b"}
Sep 09 03:24:42.130014 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.129912    2583 scope.go:117] "RemoveContainer" containerID="0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0"
Sep 09 03:24:42.139596 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.139577    2583 scope.go:117] "RemoveContainer" containerID="0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0"
Sep 09 03:24:42.139891 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:24:42.139872    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0\": container with ID starting with 0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0 not found: ID does not exist" containerID="0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0"
Sep 09 03:24:42.139980 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.139903    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0"} err="failed to get container status \"0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0\": rpc error: code = NotFound desc = could not find container \"0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0\": container with ID starting with 0106c369e7dc32a7543a5988ebf6d710961ea246bea92b0a8f9f9d57c2b5d1a0 not found: ID does not exist"
Sep 09 03:24:42.150695 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.150673    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"]
Sep 09 03:24:42.150905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.150887    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxg9\" (UniqueName: \"kubernetes.io/projected/935983ac-efa8-46db-af2b-84ca67cfd785-kube-api-access-gvxg9\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:24:42.150961 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.150909    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/935983ac-efa8-46db-af2b-84ca67cfd785-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:24:42.150961 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.150920    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/935983ac-efa8-46db-af2b-84ca67cfd785-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:24:42.154383 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:42.154362    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4ce4c1/explicit-model-4ce4c1-kserve-5c6d99dd5-s4shd"]
Sep 09 03:24:43.009732 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:43.009687    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="935983ac-efa8-46db-af2b-84ca67cfd785" path="/var/lib/kubelet/pods/935983ac-efa8-46db-af2b-84ca67cfd785/volumes"
Sep 09 03:24:52.139046 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:52.138999    2583 patch_prober.go:28] interesting pod/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 03:24:52.139445 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:52.139062    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 03:24:54.205206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:54.205169    2583 generic.go:356] "Generic (PLEG): container finished" podID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerID="a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c" exitCode=0
Sep 09 03:24:54.205624 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:54.205247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" event={"ID":"129c7d5b-784c-4ea0-99b6-b1ddfffb8449","Type":"ContainerDied","Data":"a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c"}
Sep 09 03:24:54.205689 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:54.205675    2583 scope.go:117] "RemoveContainer" containerID="a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c"
Sep 09 03:24:54.206741 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:54.206716    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:24:54.206794 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:54.206778    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:24:54.206851 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:54.206810    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:24:55.213029 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:55.212992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" event={"ID":"129c7d5b-784c-4ea0-99b6-b1ddfffb8449","Type":"ContainerStarted","Data":"ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef"}
Sep 09 03:24:56.036906 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:56.036867    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:24:56.037195 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:56.037169    2583 patch_prober.go:28] interesting pod/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 09 03:24:56.037296 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:56.037242    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 09 03:24:57.037012 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:57.036972    2583 patch_prober.go:28] interesting pod/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 09 03:24:57.037427 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:57.037029    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 09 03:24:58.037506 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:58.037466    2583 patch_prober.go:28] interesting pod/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 09 03:24:58.038021 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:58.037525    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 09 03:24:59.037300 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:59.037262    2583 patch_prober.go:28] interesting pod/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 09 03:24:59.037492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:24:59.037328    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 09 03:25:00.037562 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:00.037519    2583 patch_prober.go:28] interesting pod/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 09 03:25:00.038001 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:00.037589    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 09 03:25:01.040319 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:01.040279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:25:01.040808 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:01.040644    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:25:01.041543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:01.041525    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:25:09.212379 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.212337    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"]
Sep 09 03:25:09.212926 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.212906    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935983ac-efa8-46db-af2b-84ca67cfd785" containerName="main"
Sep 09 03:25:09.213022 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.212929    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="935983ac-efa8-46db-af2b-84ca67cfd785" containerName="main"
Sep 09 03:25:09.213086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.213067    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="935983ac-efa8-46db-af2b-84ca67cfd785" containerName="main"
Sep 09 03:25:09.216361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.216338    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"
Sep 09 03:25:09.218759 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.218735    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:25:09.218885 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.218735    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:25:09.223581 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.223550    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"]
Sep 09 03:25:09.318923 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.318890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065b69a3-e55c-4b9f-8e3a-305dc3786da1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7\" (UID: \"065b69a3-e55c-4b9f-8e3a-305dc3786da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"
Sep 09 03:25:09.419519 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.419479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065b69a3-e55c-4b9f-8e3a-305dc3786da1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7\" (UID: \"065b69a3-e55c-4b9f-8e3a-305dc3786da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"
Sep 09 03:25:09.420229 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.420200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065b69a3-e55c-4b9f-8e3a-305dc3786da1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7\" (UID: \"065b69a3-e55c-4b9f-8e3a-305dc3786da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"
Sep 09 03:25:09.529243 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.529151    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"
Sep 09 03:25:09.805698 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:09.805670    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"]
Sep 09 03:25:09.807977 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:25:09.807931    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065b69a3_e55c_4b9f_8e3a_305dc3786da1.slice/crio-97002e769ea21f483fba6097504c8f4b8e197def0e3ff83a17840bc570010e97 WatchSource:0}: Error finding container 97002e769ea21f483fba6097504c8f4b8e197def0e3ff83a17840bc570010e97: Status 404 returned error can't find the container with id 97002e769ea21f483fba6097504c8f4b8e197def0e3ff83a17840bc570010e97
Sep 09 03:25:10.298058 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:10.298022    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7" event={"ID":"065b69a3-e55c-4b9f-8e3a-305dc3786da1","Type":"ContainerStarted","Data":"cf804c777fccf1fa8196b8a7b71536c097c39b597efe928ac58763e8503ba7b8"}
Sep 09 03:25:10.298567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:10.298083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7" event={"ID":"065b69a3-e55c-4b9f-8e3a-305dc3786da1","Type":"ContainerStarted","Data":"97002e769ea21f483fba6097504c8f4b8e197def0e3ff83a17840bc570010e97"}
Sep 09 03:25:10.314173 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:10.314126    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7" podStartSLOduration=1.31411348 podStartE2EDuration="1.31411348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:10.312073198 +0000 UTC m=+1159.826155378" watchObservedRunningTime="2026-09-09 03:25:10.31411348 +0000 UTC m=+1159.828195685"
Sep 09 03:25:11.303956 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:11.303923    2583 generic.go:356] "Generic (PLEG): container finished" podID="065b69a3-e55c-4b9f-8e3a-305dc3786da1" containerID="cf804c777fccf1fa8196b8a7b71536c097c39b597efe928ac58763e8503ba7b8" exitCode=0
Sep 09 03:25:11.304324 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:11.304001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7" event={"ID":"065b69a3-e55c-4b9f-8e3a-305dc3786da1","Type":"ContainerDied","Data":"cf804c777fccf1fa8196b8a7b71536c097c39b597efe928ac58763e8503ba7b8"}
Sep 09 03:25:12.447629 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:12.447601    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"
Sep 09 03:25:12.549870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:12.549819    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065b69a3-e55c-4b9f-8e3a-305dc3786da1-maas-api-service-ca\") pod \"065b69a3-e55c-4b9f-8e3a-305dc3786da1\" (UID: \"065b69a3-e55c-4b9f-8e3a-305dc3786da1\") "
Sep 09 03:25:12.550180 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:12.550158    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065b69a3-e55c-4b9f-8e3a-305dc3786da1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "065b69a3-e55c-4b9f-8e3a-305dc3786da1" (UID: "065b69a3-e55c-4b9f-8e3a-305dc3786da1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:12.650543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:12.650508    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065b69a3-e55c-4b9f-8e3a-305dc3786da1-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:13.317065 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:13.316963    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"
Sep 09 03:25:13.317065 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:13.316999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7" event={"ID":"065b69a3-e55c-4b9f-8e3a-305dc3786da1","Type":"ContainerDied","Data":"97002e769ea21f483fba6097504c8f4b8e197def0e3ff83a17840bc570010e97"}
Sep 09 03:25:13.317065 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:13.317047    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97002e769ea21f483fba6097504c8f4b8e197def0e3ff83a17840bc570010e97"
Sep 09 03:25:16.845619 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.845583    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"]
Sep 09 03:25:16.973195 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.973155    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"]
Sep 09 03:25:16.973603 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.973586    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065b69a3-e55c-4b9f-8e3a-305dc3786da1" containerName="revoke-keys"
Sep 09 03:25:16.973603 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.973602    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="065b69a3-e55c-4b9f-8e3a-305dc3786da1" containerName="revoke-keys"
Sep 09 03:25:16.973771 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.973692    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="065b69a3-e55c-4b9f-8e3a-305dc3786da1" containerName="revoke-keys"
Sep 09 03:25:16.977383 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.977356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"
Sep 09 03:25:16.980212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.980183    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:25:16.980212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.980189    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:25:16.990141 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:16.990115    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"]
Sep 09 03:25:17.091159 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:17.091124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b96116-dfb8-4c85-80d6-321e1dbe0ae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r\" (UID: \"b4b96116-dfb8-4c85-80d6-321e1dbe0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"
Sep 09 03:25:17.192508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:17.192472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b96116-dfb8-4c85-80d6-321e1dbe0ae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r\" (UID: \"b4b96116-dfb8-4c85-80d6-321e1dbe0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"
Sep 09 03:25:17.193248 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:17.193222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b96116-dfb8-4c85-80d6-321e1dbe0ae7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r\" (UID: \"b4b96116-dfb8-4c85-80d6-321e1dbe0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"
Sep 09 03:25:17.289905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:17.289868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"
Sep 09 03:25:17.563117 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:17.563091    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"]
Sep 09 03:25:17.565406 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:25:17.565374    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b96116_dfb8_4c85_80d6_321e1dbe0ae7.slice/crio-c9efb711e5bda5e439b284311ba807a0763c7764d464db7e4021765e47a9295d WatchSource:0}: Error finding container c9efb711e5bda5e439b284311ba807a0763c7764d464db7e4021765e47a9295d: Status 404 returned error can't find the container with id c9efb711e5bda5e439b284311ba807a0763c7764d464db7e4021765e47a9295d
Sep 09 03:25:18.343623 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:18.343590    2583 generic.go:356] "Generic (PLEG): container finished" podID="b4b96116-dfb8-4c85-80d6-321e1dbe0ae7" containerID="2cfbddbab572ccd6a559f98d795034f48c0dd311f3cf39f6528584aec6c18b67" exitCode=0
Sep 09 03:25:18.343985 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:18.343677    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r" event={"ID":"b4b96116-dfb8-4c85-80d6-321e1dbe0ae7","Type":"ContainerDied","Data":"2cfbddbab572ccd6a559f98d795034f48c0dd311f3cf39f6528584aec6c18b67"}
Sep 09 03:25:18.343985 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:18.343733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r" event={"ID":"b4b96116-dfb8-4c85-80d6-321e1dbe0ae7","Type":"ContainerStarted","Data":"c9efb711e5bda5e439b284311ba807a0763c7764d464db7e4021765e47a9295d"}
Sep 09 03:25:19.484915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:19.484893    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"
Sep 09 03:25:19.515883 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:19.515849    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b96116-dfb8-4c85-80d6-321e1dbe0ae7-maas-api-service-ca\") pod \"b4b96116-dfb8-4c85-80d6-321e1dbe0ae7\" (UID: \"b4b96116-dfb8-4c85-80d6-321e1dbe0ae7\") "
Sep 09 03:25:19.516139 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:19.516117    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b96116-dfb8-4c85-80d6-321e1dbe0ae7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4b96116-dfb8-4c85-80d6-321e1dbe0ae7" (UID: "b4b96116-dfb8-4c85-80d6-321e1dbe0ae7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:19.516367 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:19.516347    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b96116-dfb8-4c85-80d6-321e1dbe0ae7-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:20.133453 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:20.133421    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"]
Sep 09 03:25:20.136266 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:20.136237    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57aecdff-rjvd7"]
Sep 09 03:25:20.355572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:20.355540    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"
Sep 09 03:25:20.355720 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:20.355587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r" event={"ID":"b4b96116-dfb8-4c85-80d6-321e1dbe0ae7","Type":"ContainerDied","Data":"c9efb711e5bda5e439b284311ba807a0763c7764d464db7e4021765e47a9295d"}
Sep 09 03:25:20.355720 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:20.355629    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9efb711e5bda5e439b284311ba807a0763c7764d464db7e4021765e47a9295d"
Sep 09 03:25:21.009423 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:21.009390    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065b69a3-e55c-4b9f-8e3a-305dc3786da1" path="/var/lib/kubelet/pods/065b69a3-e55c-4b9f-8e3a-305dc3786da1/volumes"
Sep 09 03:25:26.560045 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:26.560011    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"]
Sep 09 03:25:26.560504 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:26.560263    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy" containerID="cri-o://ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef" gracePeriod=30
Sep 09 03:25:28.045084 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:28.045045    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"]
Sep 09 03:25:28.050284 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:28.050256    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b27dde-vp45r"]
Sep 09 03:25:29.008009 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:29.007963    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b96116-dfb8-4c85-80d6-321e1dbe0ae7" path="/var/lib/kubelet/pods/b4b96116-dfb8-4c85-80d6-321e1dbe0ae7/volumes"
Sep 09 03:25:31.043135 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.043098    2583 patch_prober.go:28] interesting pod/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:25:31.043597 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.043179    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:25:31.825250 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.825174    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:25:31.865767 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.865735    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n" podUID="2d7d064c-e30e-483c-929a-248f0d43bdf4" containerName="main" containerID="cri-o://e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702" gracePeriod=45
Sep 09 03:25:31.940063 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940031    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istiod-ca-cert\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-envoy\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940125    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-podinfo\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940151    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvz2x\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-kube-api-access-jvz2x\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-certs\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940456 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940250    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-socket\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940456 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-data\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940456 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940369    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-token\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940456 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940407    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-credential-socket\") pod \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\" (UID: \"129c7d5b-784c-4ea0-99b6-b1ddfffb8449\") "
Sep 09 03:25:31.940643 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940527    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:31.940643 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940568    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-data" (OuterVolumeSpecName: "istio-data") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:31.940643 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:31.940806 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940711    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:31.940938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940915    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:31.940938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940937    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:31.941111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940950    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:31.941111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940947    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:31.941111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.940962    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:31.942566 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.942535    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:31.942721 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.942704    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-token" (OuterVolumeSpecName: "istio-token") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:31.942881 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.942854    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:25:31.942881 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:31.942858    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-kube-api-access-jvz2x" (OuterVolumeSpecName: "kube-api-access-jvz2x") pod "129c7d5b-784c-4ea0-99b6-b1ddfffb8449" (UID: "129c7d5b-784c-4ea0-99b6-b1ddfffb8449"). InnerVolumeSpecName "kube-api-access-jvz2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:32.042503 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.042471    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.042503 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.042498    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.042503 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.042508    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.042785 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.042517    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.042785 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.042526    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvz2x\" (UniqueName: \"kubernetes.io/projected/129c7d5b-784c-4ea0-99b6-b1ddfffb8449-kube-api-access-jvz2x\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.097349 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.097324    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:25:32.244134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7d064c-e30e-483c-929a-248f0d43bdf4-tls-certs\") pod \"2d7d064c-e30e-483c-929a-248f0d43bdf4\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") "
Sep 09 03:25:32.244134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244139    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-home\") pod \"2d7d064c-e30e-483c-929a-248f0d43bdf4\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") "
Sep 09 03:25:32.244396 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-model-cache\") pod \"2d7d064c-e30e-483c-929a-248f0d43bdf4\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") "
Sep 09 03:25:32.244396 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244226    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-dshm\") pod \"2d7d064c-e30e-483c-929a-248f0d43bdf4\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") "
Sep 09 03:25:32.244396 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244266    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-tmp-dir\") pod \"2d7d064c-e30e-483c-929a-248f0d43bdf4\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") "
Sep 09 03:25:32.244396 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244355    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs5tx\" (UniqueName: \"kubernetes.io/projected/2d7d064c-e30e-483c-929a-248f0d43bdf4-kube-api-access-hs5tx\") pod \"2d7d064c-e30e-483c-929a-248f0d43bdf4\" (UID: \"2d7d064c-e30e-483c-929a-248f0d43bdf4\") "
Sep 09 03:25:32.244608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244441    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-home" (OuterVolumeSpecName: "home") pod "2d7d064c-e30e-483c-929a-248f0d43bdf4" (UID: "2d7d064c-e30e-483c-929a-248f0d43bdf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:32.244608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d7d064c-e30e-483c-929a-248f0d43bdf4" (UID: "2d7d064c-e30e-483c-929a-248f0d43bdf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:32.244608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244586    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d7d064c-e30e-483c-929a-248f0d43bdf4" (UID: "2d7d064c-e30e-483c-929a-248f0d43bdf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:32.244766 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244738    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.244766 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244755    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.244911 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.244773    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.246616 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.246588    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-dshm" (OuterVolumeSpecName: "dshm") pod "2d7d064c-e30e-483c-929a-248f0d43bdf4" (UID: "2d7d064c-e30e-483c-929a-248f0d43bdf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:32.246718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.246620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7d064c-e30e-483c-929a-248f0d43bdf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d7d064c-e30e-483c-929a-248f0d43bdf4" (UID: "2d7d064c-e30e-483c-929a-248f0d43bdf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:25:32.246718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.246632    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7d064c-e30e-483c-929a-248f0d43bdf4-kube-api-access-hs5tx" (OuterVolumeSpecName: "kube-api-access-hs5tx") pod "2d7d064c-e30e-483c-929a-248f0d43bdf4" (UID: "2d7d064c-e30e-483c-929a-248f0d43bdf4"). InnerVolumeSpecName "kube-api-access-hs5tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:32.346205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.346115    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7d064c-e30e-483c-929a-248f0d43bdf4-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.346205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.346146    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5tx\" (UniqueName: \"kubernetes.io/projected/2d7d064c-e30e-483c-929a-248f0d43bdf4-kube-api-access-hs5tx\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.346205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.346156    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7d064c-e30e-483c-929a-248f0d43bdf4-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:32.423225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.423183    2583 generic.go:356] "Generic (PLEG): container finished" podID="2d7d064c-e30e-483c-929a-248f0d43bdf4" containerID="e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702" exitCode=0
Sep 09 03:25:32.423424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.423238    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"
Sep 09 03:25:32.423424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.423257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n" event={"ID":"2d7d064c-e30e-483c-929a-248f0d43bdf4","Type":"ContainerDied","Data":"e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702"}
Sep 09 03:25:32.423424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.423301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n" event={"ID":"2d7d064c-e30e-483c-929a-248f0d43bdf4","Type":"ContainerDied","Data":"914ad2ef89303f4cb7cbd82675a0cd3e9f7620c408b29f4cbb5cf33ee3c45f18"}
Sep 09 03:25:32.423424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.423321    2583 scope.go:117] "RemoveContainer" containerID="e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702"
Sep 09 03:25:32.425489 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.425457    2583 generic.go:356] "Generic (PLEG): container finished" podID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerID="ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef" exitCode=0
Sep 09 03:25:32.425604 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.425511    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" event={"ID":"129c7d5b-784c-4ea0-99b6-b1ddfffb8449","Type":"ContainerDied","Data":"ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef"}
Sep 09 03:25:32.425604 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.425529    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"
Sep 09 03:25:32.425604 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.425556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct" event={"ID":"129c7d5b-784c-4ea0-99b6-b1ddfffb8449","Type":"ContainerDied","Data":"2fbb34032d2991c94bb991825ee40a08e0d0f1f0767421b86a8da58e924b75b2"}
Sep 09 03:25:32.434311 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.434291    2583 scope.go:117] "RemoveContainer" containerID="e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702"
Sep 09 03:25:32.434569 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:25:32.434552    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702\": container with ID starting with e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702 not found: ID does not exist" containerID="e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702"
Sep 09 03:25:32.434635 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.434578    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702"} err="failed to get container status \"e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702\": rpc error: code = NotFound desc = could not find container \"e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702\": container with ID starting with e2af52756c932395a0e41154eaa4d21048f2526168bbb7f41b6b3ae840b15702 not found: ID does not exist"
Sep 09 03:25:32.434635 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.434592    2583 scope.go:117] "RemoveContainer" containerID="ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef"
Sep 09 03:25:32.443174 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.443151    2583 scope.go:117] "RemoveContainer" containerID="a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c"
Sep 09 03:25:32.450516 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.450494    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"]
Sep 09 03:25:32.452192 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.452178    2583 scope.go:117] "RemoveContainer" containerID="ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef"
Sep 09 03:25:32.452458 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:25:32.452442    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef\": container with ID starting with ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef not found: ID does not exist" containerID="ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef"
Sep 09 03:25:32.452502 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.452464    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef"} err="failed to get container status \"ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef\": rpc error: code = NotFound desc = could not find container \"ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef\": container with ID starting with ff3bdb2f25a5385ecc0fd640e2a3eb79b5815c3f15633e65586669a2391764ef not found: ID does not exist"
Sep 09 03:25:32.452502 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.452481    2583 scope.go:117] "RemoveContainer" containerID="a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c"
Sep 09 03:25:32.452694 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:25:32.452677    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c\": container with ID starting with a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c not found: ID does not exist" containerID="a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c"
Sep 09 03:25:32.452735 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.452702    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c"} err="failed to get container status \"a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c\": rpc error: code = NotFound desc = could not find container \"a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c\": container with ID starting with a170979cb031ac304dbbe984ea8ca091ec64f86850fc8cd4fd489d23bb4a995c not found: ID does not exist"
Sep 09 03:25:32.457388 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.457366    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-70784b/test-model-70784b-kserve-6ff8975589-ltx2n"]
Sep 09 03:25:32.470337 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.470309    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"]
Sep 09 03:25:32.477614 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:32.477592    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-57aecdff-openshift-default-5f5859d496-89dct"]
Sep 09 03:25:33.008348 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:33.008304    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" path="/var/lib/kubelet/pods/129c7d5b-784c-4ea0-99b6-b1ddfffb8449/volumes"
Sep 09 03:25:33.008727 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:33.008714    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7d064c-e30e-483c-929a-248f0d43bdf4" path="/var/lib/kubelet/pods/2d7d064c-e30e-483c-929a-248f0d43bdf4/volumes"
Sep 09 03:25:50.976502 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:50.976465    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:25:50.977938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:50.977904    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:25:50.983275 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:50.983257    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:25:50.984565 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:50.984548    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:25:53.458590 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:53.458557    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"]
Sep 09 03:25:53.459013 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:53.458760    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" containerID="cri-o://adb8e685fc8cd21ab1988cd7062f45f991701e42f67544253c44bea87143ec42" gracePeriod=30
Sep 09 03:25:58.018173 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.018137    2583 patch_prober.go:28] interesting pod/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:25:58.018539 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.018193    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:25:58.583361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.583321    2583 generic.go:356] "Generic (PLEG): container finished" podID="20920c8d-f6ec-473f-936f-c63d27834653" containerID="adb8e685fc8cd21ab1988cd7062f45f991701e42f67544253c44bea87143ec42" exitCode=0
Sep 09 03:25:58.583573 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.583386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" event={"ID":"20920c8d-f6ec-473f-936f-c63d27834653","Type":"ContainerDied","Data":"adb8e685fc8cd21ab1988cd7062f45f991701e42f67544253c44bea87143ec42"}
Sep 09 03:25:58.583573 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.583467    2583 scope.go:117] "RemoveContainer" containerID="d343bd88d002b8e14a7c257c6352e92d871385337ea8988514ca05bb92ea1415"
Sep 09 03:25:58.712492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.712467    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:25:58.811786 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.811697    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccg9g\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-kube-api-access-ccg9g\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.811786 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.811738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-istio-token\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.811786 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.811766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20920c8d-f6ec-473f-936f-c63d27834653-istiod-ca-cert\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.811786 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.811786    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-certs\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.812155 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.811819    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20920c8d-f6ec-473f-936f-c63d27834653-istio-podinfo\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.812155 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.811872    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-socket\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.812155 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.811911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-credential-socket\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.812155 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.811972    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-data\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.812155 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812023    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-envoy\") pod \"20920c8d-f6ec-473f-936f-c63d27834653\" (UID: \"20920c8d-f6ec-473f-936f-c63d27834653\") "
Sep 09 03:25:58.812418 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812212    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20920c8d-f6ec-473f-936f-c63d27834653-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:25:58.812418 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812221    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:58.812418 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812209    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:58.812735 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812434    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:58.812735 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812459    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20920c8d-f6ec-473f-936f-c63d27834653-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:58.812735 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812478    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:58.812735 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812490    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:58.812962 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.812797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-data" (OuterVolumeSpecName: "istio-data") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:58.814263 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.814233    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-kube-api-access-ccg9g" (OuterVolumeSpecName: "kube-api-access-ccg9g") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "kube-api-access-ccg9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:58.814374 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.814281    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20920c8d-f6ec-473f-936f-c63d27834653-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:25:58.814374 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.814289    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-istio-token" (OuterVolumeSpecName: "istio-token") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:58.814498 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.814476    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20920c8d-f6ec-473f-936f-c63d27834653" (UID: "20920c8d-f6ec-473f-936f-c63d27834653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:25:58.913959 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.913921    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20920c8d-f6ec-473f-936f-c63d27834653-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:58.913959 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.913952    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:58.913959 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.913965    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:58.914212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.913977    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20920c8d-f6ec-473f-936f-c63d27834653-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:58.914212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.913988    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccg9g\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-kube-api-access-ccg9g\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:58.914212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:58.914001    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20920c8d-f6ec-473f-936f-c63d27834653-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:25:59.590009 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:59.589981    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"
Sep 09 03:25:59.590009 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:59.589993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9" event={"ID":"20920c8d-f6ec-473f-936f-c63d27834653","Type":"ContainerDied","Data":"7c90be2490ab4dc85ec494e786f7ece503459a448207fc8c94dc5a75c9ed9cd6"}
Sep 09 03:25:59.590547 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:59.590060    2583 scope.go:117] "RemoveContainer" containerID="adb8e685fc8cd21ab1988cd7062f45f991701e42f67544253c44bea87143ec42"
Sep 09 03:25:59.612647 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:59.612617    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"]
Sep 09 03:25:59.626876 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:25:59.626846    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-70784b-openshift-default-799dfdf49b-frgs9"]
Sep 09 03:26:01.007271 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:01.007238    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20920c8d-f6ec-473f-936f-c63d27834653" path="/var/lib/kubelet/pods/20920c8d-f6ec-473f-936f-c63d27834653/volumes"
Sep 09 03:26:08.895977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.895937    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"]
Sep 09 03:26:08.896541 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896521    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy"
Sep 09 03:26:08.896632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896545    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy"
Sep 09 03:26:08.896632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896567    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy"
Sep 09 03:26:08.896632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896575    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy"
Sep 09 03:26:08.896632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896584    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b96116-dfb8-4c85-80d6-321e1dbe0ae7" containerName="revoke-keys"
Sep 09 03:26:08.896632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896592    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b96116-dfb8-4c85-80d6-321e1dbe0ae7" containerName="revoke-keys"
Sep 09 03:26:08.896632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896608    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7d064c-e30e-483c-929a-248f0d43bdf4" containerName="main"
Sep 09 03:26:08.896632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896617    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7d064c-e30e-483c-929a-248f0d43bdf4" containerName="main"
Sep 09 03:26:08.896632 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896628    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896636    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896648    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896655    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896767    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b96116-dfb8-4c85-80d6-321e1dbe0ae7" containerName="revoke-keys"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896780    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896791    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20920c8d-f6ec-473f-936f-c63d27834653" containerName="istio-proxy"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896802    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7d064c-e30e-483c-929a-248f0d43bdf4" containerName="main"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896810    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy"
Sep 09 03:26:08.897064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.896818    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="129c7d5b-784c-4ea0-99b6-b1ddfffb8449" containerName="istio-proxy"
Sep 09 03:26:08.901368 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.901345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:08.904333 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.904310    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4c8420-openshift-default-dockercfg-r787p\""
Sep 09 03:26:08.918207 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:08.918177    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"]
Sep 09 03:26:09.007846 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.007809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgnvr\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-kube-api-access-dgnvr\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.008052 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.007876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fed7303-16c7-472c-a928-da7dc6441174-istiod-ca-cert\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.008052 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.007905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-envoy\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.008052 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.007934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-data\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.008227 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.008045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-certs\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.008227 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.008122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-istio-token\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.008227 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.008160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-credential-socket\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.008227 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.008205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-socket\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.008375 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.008235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fed7303-16c7-472c-a928-da7dc6441174-istio-podinfo\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.109404 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-istio-token\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.109404 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-credential-socket\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.109668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-socket\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.109668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fed7303-16c7-472c-a928-da7dc6441174-istio-podinfo\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.109668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgnvr\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-kube-api-access-dgnvr\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.109668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fed7303-16c7-472c-a928-da7dc6441174-istiod-ca-cert\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.109668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-envoy\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.109668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-data\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.110017 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-certs\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.110017 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.109882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-credential-socket\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.110132 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.110037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-certs\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.110132 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.110057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-data\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.110231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.110176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-socket\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.110272 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.110244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fed7303-16c7-472c-a928-da7dc6441174-istiod-ca-cert\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.112111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.112090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-envoy\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.112318 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.112299    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fed7303-16c7-472c-a928-da7dc6441174-istio-podinfo\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.118551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.118526    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-istio-token\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.118874 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.118857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgnvr\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-kube-api-access-dgnvr\") pod \"e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") " pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.212886 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.212774    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:09.489307 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.489279    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"]
Sep 09 03:26:09.491870 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:26:09.491814    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fed7303_16c7_472c_a928_da7dc6441174.slice/crio-64236f315c66de4ef4a2f9767422a75591175ce565902a7eee39701818d4f8ef WatchSource:0}: Error finding container 64236f315c66de4ef4a2f9767422a75591175ce565902a7eee39701818d4f8ef: Status 404 returned error can't find the container with id 64236f315c66de4ef4a2f9767422a75591175ce565902a7eee39701818d4f8ef
Sep 09 03:26:09.494377 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.494337    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:09.494448 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.494430    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:09.494492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.494477    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:09.649620 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.649582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" event={"ID":"8fed7303-16c7-472c-a928-da7dc6441174","Type":"ContainerStarted","Data":"ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29"}
Sep 09 03:26:09.649820 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.649649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" event={"ID":"8fed7303-16c7-472c-a928-da7dc6441174","Type":"ContainerStarted","Data":"64236f315c66de4ef4a2f9767422a75591175ce565902a7eee39701818d4f8ef"}
Sep 09 03:26:09.672333 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:09.672279    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" podStartSLOduration=1.6722627989999999 podStartE2EDuration="1.672262799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:09.670323363 +0000 UTC m=+1219.184405541" watchObservedRunningTime="2026-09-09 03:26:09.672262799 +0000 UTC m=+1219.186344950"
Sep 09 03:26:10.213236 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:10.213204    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:10.219082 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:10.219055    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:10.655006 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:10.654968    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:10.655996 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:10.655973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:23.390544 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.390502    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"]
Sep 09 03:26:23.394655 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.394629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.397917 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.397895    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-cd1da6-openshift-default-dockercfg-x5bln\""
Sep 09 03:26:23.418392 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.418367    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"]
Sep 09 03:26:23.444782 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.444744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c67ea8-e39e-499c-965a-369909fd1478-istiod-ca-cert\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.444993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.444794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-certs\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.444993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.444895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c67ea8-e39e-499c-965a-369909fd1478-istio-podinfo\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.444993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.444922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkxs\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-kube-api-access-vpkxs\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.444993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.444942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-socket\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.444993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.444968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-envoy\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.445187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.445005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-credential-socket\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.445187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.445038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-istio-token\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.445187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.445096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-data\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.545876 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.545810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c67ea8-e39e-499c-965a-369909fd1478-istiod-ca-cert\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.545895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-certs\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.545922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c67ea8-e39e-499c-965a-369909fd1478-istio-podinfo\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.545938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkxs\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-kube-api-access-vpkxs\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.545956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-socket\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.545985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-envoy\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546073 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.546013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-credential-socket\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546348 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.546324    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-istio-token\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546405 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.546366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-credential-socket\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546462 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.546410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-certs\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546462 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.546450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-data\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546575 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.546534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-socket\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546705 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.546679    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c67ea8-e39e-499c-965a-369909fd1478-istiod-ca-cert\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.546763 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.546722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-data\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.548594 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.548566    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-envoy\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.548780 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.548765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c67ea8-e39e-499c-965a-369909fd1478-istio-podinfo\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.561175 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.561148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-istio-token\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.562164 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.562145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkxs\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-kube-api-access-vpkxs\") pod \"e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") " pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:23.704881 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:23.704776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:24.016308 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:24.016221    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"]
Sep 09 03:26:24.019774 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:26:24.019742    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c67ea8_e39e_499c_965a_369909fd1478.slice/crio-788cebda89db8c3785ce1bb2c4afd22979bcb5b54667227e108bc242ebfc8622 WatchSource:0}: Error finding container 788cebda89db8c3785ce1bb2c4afd22979bcb5b54667227e108bc242ebfc8622: Status 404 returned error can't find the container with id 788cebda89db8c3785ce1bb2c4afd22979bcb5b54667227e108bc242ebfc8622
Sep 09 03:26:24.022068 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:24.022032    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:24.022188 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:24.022120    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:24.022188 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:24.022167    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:24.730923 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:24.730883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" event={"ID":"29c67ea8-e39e-499c-965a-369909fd1478","Type":"ContainerStarted","Data":"f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898"}
Sep 09 03:26:24.731335 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:24.730970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" event={"ID":"29c67ea8-e39e-499c-965a-369909fd1478","Type":"ContainerStarted","Data":"788cebda89db8c3785ce1bb2c4afd22979bcb5b54667227e108bc242ebfc8622"}
Sep 09 03:26:24.774072 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:24.774005    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podStartSLOduration=1.773968513 podStartE2EDuration="1.773968513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:24.773050521 +0000 UTC m=+1234.287132697" watchObservedRunningTime="2026-09-09 03:26:24.773968513 +0000 UTC m=+1234.288050682"
Sep 09 03:26:25.705176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:25.705131    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:25.710624 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:25.710591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:25.737492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:25.737464    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:25.738474 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:25.738455    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:30.765290 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:30.765255    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fed7303-16c7-472c-a928-da7dc6441174" containerID="ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29" exitCode=0
Sep 09 03:26:30.765708 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:30.765328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" event={"ID":"8fed7303-16c7-472c-a928-da7dc6441174","Type":"ContainerDied","Data":"ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29"}
Sep 09 03:26:30.765775 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:30.765761    2583 scope.go:117] "RemoveContainer" containerID="ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29"
Sep 09 03:26:30.766935 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:30.766910    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:30.767001 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:30.766979    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:30.767041 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:30.767014    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:31.776266 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:31.776220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" event={"ID":"8fed7303-16c7-472c-a928-da7dc6441174","Type":"ContainerStarted","Data":"ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3"}
Sep 09 03:26:32.213778 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:32.213736    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:32.214056 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:32.214001    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:26:32.214106 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:32.214049    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:26:33.213381 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:33.213335    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:26:33.213790 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:33.213413    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:26:34.214128 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:34.214088    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:26:34.214572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:34.214156    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:26:35.213495 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:35.213437    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:26:35.213729 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:35.213517    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:26:36.213933 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:36.213888    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:26:36.214406 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:36.214383    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:26:37.162971 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.162932    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:37.165749 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.165711    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:37.169646 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.168870    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-49nx9\""
Sep 09 03:26:37.176713 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.176678    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:37.272859 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.265057    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:37.272859 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.266745    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:37.276207 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.276174    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:26:37.305599 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.305559    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgn8x\" (UniqueName: \"kubernetes.io/projected/09dfed8b-4bd7-464e-a401-8980186fd336-kube-api-access-bgn8x\") pod \"test-curl-7601-140566200977568\" (UID: \"09dfed8b-4bd7-464e-a401-8980186fd336\") " pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:37.407049 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.407007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgn8x\" (UniqueName: \"kubernetes.io/projected/09dfed8b-4bd7-464e-a401-8980186fd336-kube-api-access-bgn8x\") pod \"test-curl-7601-140566200977568\" (UID: \"09dfed8b-4bd7-464e-a401-8980186fd336\") " pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:37.416265 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.416194    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgn8x\" (UniqueName: \"kubernetes.io/projected/09dfed8b-4bd7-464e-a401-8980186fd336-kube-api-access-bgn8x\") pod \"test-curl-7601-140566200977568\" (UID: \"09dfed8b-4bd7-464e-a401-8980186fd336\") " pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:37.480539 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.480499    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:37.756128 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.756097    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:37.758741 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:26:37.758708    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09dfed8b_4bd7_464e_a401_8980186fd336.slice/crio-d7be8b0390e525535b9b0ac7ebd30fd4384dd03c18e7d057f1bef50d335e291f WatchSource:0}: Error finding container d7be8b0390e525535b9b0ac7ebd30fd4384dd03c18e7d057f1bef50d335e291f: Status 404 returned error can't find the container with id d7be8b0390e525535b9b0ac7ebd30fd4384dd03c18e7d057f1bef50d335e291f
Sep 09 03:26:37.760273 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.760252    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:26:37.823824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:37.823788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7601-140566200977568" event={"ID":"09dfed8b-4bd7-464e-a401-8980186fd336","Type":"ContainerStarted","Data":"d7be8b0390e525535b9b0ac7ebd30fd4384dd03c18e7d057f1bef50d335e291f"}
Sep 09 03:26:38.831185 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:38.831148    2583 generic.go:356] "Generic (PLEG): container finished" podID="09dfed8b-4bd7-464e-a401-8980186fd336" containerID="4124bac1ad74690a843a2075b651d4881001290244d437d8de90f644358030f7" exitCode=0
Sep 09 03:26:38.831636 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:38.831237    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7601-140566200977568" event={"ID":"09dfed8b-4bd7-464e-a401-8980186fd336","Type":"ContainerDied","Data":"4124bac1ad74690a843a2075b651d4881001290244d437d8de90f644358030f7"}
Sep 09 03:26:40.014582 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.014553    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:40.051411 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.051382    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7601-140566200977568_09dfed8b-4bd7-464e-a401-8980186fd336/test-curl-7601-140566200977568/0.log"
Sep 09 03:26:40.076304 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.076266    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:40.080255 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.080229    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:40.133809 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.133773    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgn8x\" (UniqueName: \"kubernetes.io/projected/09dfed8b-4bd7-464e-a401-8980186fd336-kube-api-access-bgn8x\") pod \"09dfed8b-4bd7-464e-a401-8980186fd336\" (UID: \"09dfed8b-4bd7-464e-a401-8980186fd336\") "
Sep 09 03:26:40.136154 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.136116    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dfed8b-4bd7-464e-a401-8980186fd336-kube-api-access-bgn8x" (OuterVolumeSpecName: "kube-api-access-bgn8x") pod "09dfed8b-4bd7-464e-a401-8980186fd336" (UID: "09dfed8b-4bd7-464e-a401-8980186fd336"). InnerVolumeSpecName "kube-api-access-bgn8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:26:40.203716 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.203674    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:40.204202 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.204182    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09dfed8b-4bd7-464e-a401-8980186fd336" containerName="test-curl-7601-140566200977568"
Sep 09 03:26:40.204316 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.204203    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09dfed8b-4bd7-464e-a401-8980186fd336" containerName="test-curl-7601-140566200977568"
Sep 09 03:26:40.204377 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.204312    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09dfed8b-4bd7-464e-a401-8980186fd336" containerName="test-curl-7601-140566200977568"
Sep 09 03:26:40.206350 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.206329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:40.223922 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.223149    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:40.234588 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.234551    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgn8x\" (UniqueName: \"kubernetes.io/projected/09dfed8b-4bd7-464e-a401-8980186fd336-kube-api-access-bgn8x\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:26:40.335798 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.335757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxx6q\" (UniqueName: \"kubernetes.io/projected/2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc-kube-api-access-xxx6q\") pod \"test-curl-7601-140566200977568\" (UID: \"2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc\") " pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:40.436433 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.436333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxx6q\" (UniqueName: \"kubernetes.io/projected/2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc-kube-api-access-xxx6q\") pod \"test-curl-7601-140566200977568\" (UID: \"2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc\") " pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:40.445568 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.445531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxx6q\" (UniqueName: \"kubernetes.io/projected/2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc-kube-api-access-xxx6q\") pod \"test-curl-7601-140566200977568\" (UID: \"2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc\") " pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:40.527493 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.527429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:40.810657 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.810631    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:40.813185 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:26:40.813152    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e6fbc80_ef93_4fa4_af6c_a57412aa7ccc.slice/crio-8c79d4219cab0d39f6f0f1e34fa33a7ad377b413804cf4e212216c6890de51d3 WatchSource:0}: Error finding container 8c79d4219cab0d39f6f0f1e34fa33a7ad377b413804cf4e212216c6890de51d3: Status 404 returned error can't find the container with id 8c79d4219cab0d39f6f0f1e34fa33a7ad377b413804cf4e212216c6890de51d3
Sep 09 03:26:40.844197 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.844168    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:40.844384 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.844204    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7be8b0390e525535b9b0ac7ebd30fd4384dd03c18e7d057f1bef50d335e291f"
Sep 09 03:26:40.845793 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.845757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7601-140566200977568" event={"ID":"2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc","Type":"ContainerStarted","Data":"8c79d4219cab0d39f6f0f1e34fa33a7ad377b413804cf4e212216c6890de51d3"}
Sep 09 03:26:40.856872 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:40.856823    2583 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7601-140566200977568" oldPodUID="09dfed8b-4bd7-464e-a401-8980186fd336" podUID="2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc"
Sep 09 03:26:41.008959 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:41.008927    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09dfed8b-4bd7-464e-a401-8980186fd336" path="/var/lib/kubelet/pods/09dfed8b-4bd7-464e-a401-8980186fd336/volumes"
Sep 09 03:26:41.780036 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:41.779988    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 03:26:41.780449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:41.780062    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 03:26:41.853410 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:41.853374    2583 generic.go:356] "Generic (PLEG): container finished" podID="2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc" containerID="bd7156dd5f6c8d68f2736958f3ff5da61e6119a1339a328a2486434594b37527" exitCode=0
Sep 09 03:26:41.853581 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:41.853464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7601-140566200977568" event={"ID":"2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc","Type":"ContainerDied","Data":"bd7156dd5f6c8d68f2736958f3ff5da61e6119a1339a328a2486434594b37527"}
Sep 09 03:26:43.022806 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.022774    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:43.046524 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.046481    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7601-140566200977568_2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc/test-curl-7601-140566200977568/0.log"
Sep 09 03:26:43.071365 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.071333    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:43.075176 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.075142    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7601-140566200977568"]
Sep 09 03:26:43.164477 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.164444    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxx6q\" (UniqueName: \"kubernetes.io/projected/2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc-kube-api-access-xxx6q\") pod \"2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc\" (UID: \"2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc\") "
Sep 09 03:26:43.166643 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.166614    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc-kube-api-access-xxx6q" (OuterVolumeSpecName: "kube-api-access-xxx6q") pod "2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc" (UID: "2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc"). InnerVolumeSpecName "kube-api-access-xxx6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:26:43.191272 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.191240    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:43.191622 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.191610    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc" containerName="test-curl-7601-140566200977568"
Sep 09 03:26:43.191678 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.191625    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc" containerName="test-curl-7601-140566200977568"
Sep 09 03:26:43.191719 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.191704    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc" containerName="test-curl-7601-140566200977568"
Sep 09 03:26:43.194340 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.194325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:43.203824 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.203799    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:43.266273 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.266236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7q59\" (UniqueName: \"kubernetes.io/projected/3f621049-041e-4c43-9052-3b9d6ca07735-kube-api-access-t7q59\") pod \"test-curl-7601-140566200978144\" (UID: \"3f621049-041e-4c43-9052-3b9d6ca07735\") " pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:43.266462 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.266361    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxx6q\" (UniqueName: \"kubernetes.io/projected/2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc-kube-api-access-xxx6q\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:26:43.367371 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.367343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7q59\" (UniqueName: \"kubernetes.io/projected/3f621049-041e-4c43-9052-3b9d6ca07735-kube-api-access-t7q59\") pod \"test-curl-7601-140566200978144\" (UID: \"3f621049-041e-4c43-9052-3b9d6ca07735\") " pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:43.376855 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.376800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7q59\" (UniqueName: \"kubernetes.io/projected/3f621049-041e-4c43-9052-3b9d6ca07735-kube-api-access-t7q59\") pod \"test-curl-7601-140566200978144\" (UID: \"3f621049-041e-4c43-9052-3b9d6ca07735\") " pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:43.503462 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.503418    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:43.773910 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.773813    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:43.776193 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:26:43.776164    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f621049_041e_4c43_9052_3b9d6ca07735.slice/crio-07b86df842e6e120fee188f87f5461b6ede97cb55d4c4ec309c1c3e7c399d8e9 WatchSource:0}: Error finding container 07b86df842e6e120fee188f87f5461b6ede97cb55d4c4ec309c1c3e7c399d8e9: Status 404 returned error can't find the container with id 07b86df842e6e120fee188f87f5461b6ede97cb55d4c4ec309c1c3e7c399d8e9
Sep 09 03:26:43.865840 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.865800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7601-140566200978144" event={"ID":"3f621049-041e-4c43-9052-3b9d6ca07735","Type":"ContainerStarted","Data":"07b86df842e6e120fee188f87f5461b6ede97cb55d4c4ec309c1c3e7c399d8e9"}
Sep 09 03:26:43.867030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.867011    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:43.867139 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.867039    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c79d4219cab0d39f6f0f1e34fa33a7ad377b413804cf4e212216c6890de51d3"
Sep 09 03:26:43.868768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.868743    2583 generic.go:356] "Generic (PLEG): container finished" podID="29c67ea8-e39e-499c-965a-369909fd1478" containerID="f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898" exitCode=0
Sep 09 03:26:43.868768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.868769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" event={"ID":"29c67ea8-e39e-499c-965a-369909fd1478","Type":"ContainerDied","Data":"f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898"}
Sep 09 03:26:43.869170 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.869157    2583 scope.go:117] "RemoveContainer" containerID="f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898"
Sep 09 03:26:43.870415 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.870381    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:43.870497 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.870479    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:43.870547 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:43.870533    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:43.892574 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:26:43.892546    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7601-140566200977568\" is forbidden: User \"system:node:ip-10-0-143-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-233.ec2.internal' and this object" podUID="2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:44.896930 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:44.896893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" event={"ID":"29c67ea8-e39e-499c-965a-369909fd1478","Type":"ContainerStarted","Data":"ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61"}
Sep 09 03:26:44.899570 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:26:44.899536    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7601-140566200977568\" is forbidden: User \"system:node:ip-10-0-143-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-233.ec2.internal' and this object" podUID="2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc" pod="opendatahub/test-curl-7601-140566200977568"
Sep 09 03:26:45.008963 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:45.008932    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc" path="/var/lib/kubelet/pods/2e6fbc80-ef93-4fa4-af6c-a57412aa7ccc/volumes"
Sep 09 03:26:45.705393 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:45.705354    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:45.705599 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:45.705579    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 03:26:45.705677 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:45.705639    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 03:26:45.912086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:45.912050    2583 generic.go:356] "Generic (PLEG): container finished" podID="3f621049-041e-4c43-9052-3b9d6ca07735" containerID="7761f23463cd9c0dfd8f2db556ff39e3c111de34b000fc507bb28167378f8972" exitCode=0
Sep 09 03:26:45.912554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:45.912120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7601-140566200978144" event={"ID":"3f621049-041e-4c43-9052-3b9d6ca07735","Type":"ContainerDied","Data":"7761f23463cd9c0dfd8f2db556ff39e3c111de34b000fc507bb28167378f8972"}
Sep 09 03:26:46.705728 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:46.705689    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 03:26:46.705935 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:46.705752    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 03:26:47.101186 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.101160    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:47.126007 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.125960    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7601-140566200978144_3f621049-041e-4c43-9052-3b9d6ca07735/test-curl-7601-140566200978144/0.log"
Sep 09 03:26:47.149994 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.149962    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:47.155661 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.155637    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:47.206549 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.206507    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7q59\" (UniqueName: \"kubernetes.io/projected/3f621049-041e-4c43-9052-3b9d6ca07735-kube-api-access-t7q59\") pod \"3f621049-041e-4c43-9052-3b9d6ca07735\" (UID: \"3f621049-041e-4c43-9052-3b9d6ca07735\") "
Sep 09 03:26:47.208808 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.208781    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f621049-041e-4c43-9052-3b9d6ca07735-kube-api-access-t7q59" (OuterVolumeSpecName: "kube-api-access-t7q59") pod "3f621049-041e-4c43-9052-3b9d6ca07735" (UID: "3f621049-041e-4c43-9052-3b9d6ca07735"). InnerVolumeSpecName "kube-api-access-t7q59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:26:47.284706 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.284601    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:47.285343 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.285301    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f621049-041e-4c43-9052-3b9d6ca07735" containerName="test-curl-7601-140566200978144"
Sep 09 03:26:47.285466 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.285346    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f621049-041e-4c43-9052-3b9d6ca07735" containerName="test-curl-7601-140566200978144"
Sep 09 03:26:47.285545 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.285529    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f621049-041e-4c43-9052-3b9d6ca07735" containerName="test-curl-7601-140566200978144"
Sep 09 03:26:47.288480 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.288451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:47.292429 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.292396    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:47.307608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.307568    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7q59\" (UniqueName: \"kubernetes.io/projected/3f621049-041e-4c43-9052-3b9d6ca07735-kube-api-access-t7q59\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:26:47.408539 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.408497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8ps\" (UniqueName: \"kubernetes.io/projected/4d8912a7-9f4f-4475-8e81-eefe8b49cfc7-kube-api-access-qm8ps\") pod \"test-curl-7601-140566200978144\" (UID: \"4d8912a7-9f4f-4475-8e81-eefe8b49cfc7\") " pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:47.510133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.510099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8ps\" (UniqueName: \"kubernetes.io/projected/4d8912a7-9f4f-4475-8e81-eefe8b49cfc7-kube-api-access-qm8ps\") pod \"test-curl-7601-140566200978144\" (UID: \"4d8912a7-9f4f-4475-8e81-eefe8b49cfc7\") " pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:47.518480 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.518449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8ps\" (UniqueName: \"kubernetes.io/projected/4d8912a7-9f4f-4475-8e81-eefe8b49cfc7-kube-api-access-qm8ps\") pod \"test-curl-7601-140566200978144\" (UID: \"4d8912a7-9f4f-4475-8e81-eefe8b49cfc7\") " pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:47.600784 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.600699    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:47.705602 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.705567    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 03:26:47.705775 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.705627    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 03:26:47.875352 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.875309    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:47.925421 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.925391    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:47.925596 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.925432    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b86df842e6e120fee188f87f5461b6ede97cb55d4c4ec309c1c3e7c399d8e9"
Sep 09 03:26:47.926730 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.926706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7601-140566200978144" event={"ID":"4d8912a7-9f4f-4475-8e81-eefe8b49cfc7","Type":"ContainerStarted","Data":"5b8735b507eb6818d60d5a00969a7bcabbb244ef0bec29e9829b2acf3ea6b4a6"}
Sep 09 03:26:47.937530 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:47.937508    2583 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7601-140566200978144" oldPodUID="3f621049-041e-4c43-9052-3b9d6ca07735" podUID="4d8912a7-9f4f-4475-8e81-eefe8b49cfc7"
Sep 09 03:26:48.705845 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:48.705742    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 03:26:48.705845 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:48.705803    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 03:26:48.934390 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:48.934353    2583 generic.go:356] "Generic (PLEG): container finished" podID="4d8912a7-9f4f-4475-8e81-eefe8b49cfc7" containerID="b8b64c9a25a0436a91ea55dcd628c79f720f25804159c0b03567db2ab475c6cd" exitCode=0
Sep 09 03:26:48.934540 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:48.934438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7601-140566200978144" event={"ID":"4d8912a7-9f4f-4475-8e81-eefe8b49cfc7","Type":"ContainerDied","Data":"b8b64c9a25a0436a91ea55dcd628c79f720f25804159c0b03567db2ab475c6cd"}
Sep 09 03:26:49.007888 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:49.007786    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f621049-041e-4c43-9052-3b9d6ca07735" path="/var/lib/kubelet/pods/3f621049-041e-4c43-9052-3b9d6ca07735/volumes"
Sep 09 03:26:49.706192 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:49.706156    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 03:26:49.706607 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:49.706217    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 03:26:50.124847 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.124794    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:50.156391 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.156363    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7601-140566200978144_4d8912a7-9f4f-4475-8e81-eefe8b49cfc7/test-curl-7601-140566200978144/0.log"
Sep 09 03:26:50.186658 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.186624    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:50.190993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.190963    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7601-140566200978144"]
Sep 09 03:26:50.238439 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.238397    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm8ps\" (UniqueName: \"kubernetes.io/projected/4d8912a7-9f4f-4475-8e81-eefe8b49cfc7-kube-api-access-qm8ps\") pod \"4d8912a7-9f4f-4475-8e81-eefe8b49cfc7\" (UID: \"4d8912a7-9f4f-4475-8e81-eefe8b49cfc7\") "
Sep 09 03:26:50.240934 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.240882    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8912a7-9f4f-4475-8e81-eefe8b49cfc7-kube-api-access-qm8ps" (OuterVolumeSpecName: "kube-api-access-qm8ps") pod "4d8912a7-9f4f-4475-8e81-eefe8b49cfc7" (UID: "4d8912a7-9f4f-4475-8e81-eefe8b49cfc7"). InnerVolumeSpecName "kube-api-access-qm8ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:26:50.339293 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.339192    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm8ps\" (UniqueName: \"kubernetes.io/projected/4d8912a7-9f4f-4475-8e81-eefe8b49cfc7-kube-api-access-qm8ps\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:26:50.709193 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.709150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:50.709567 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.709533    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:50.710338 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.710323    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:26:50.952620 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.952590    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:50.952799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:50.952629    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b8735b507eb6818d60d5a00969a7bcabbb244ef0bec29e9829b2acf3ea6b4a6"
Sep 09 03:26:50.963000 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:26:50.962976    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7601-140566200978144\" is forbidden: User \"system:node:ip-10-0-143-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-233.ec2.internal' and this object" podUID="4d8912a7-9f4f-4475-8e81-eefe8b49cfc7" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:51.006216 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:26:51.006170    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7601-140566200978144\" is forbidden: User \"system:node:ip-10-0-143-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-233.ec2.internal' and this object" podUID="4d8912a7-9f4f-4475-8e81-eefe8b49cfc7" pod="opendatahub/test-curl-7601-140566200978144"
Sep 09 03:26:51.008459 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.008429    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8912a7-9f4f-4475-8e81-eefe8b49cfc7" path="/var/lib/kubelet/pods/4d8912a7-9f4f-4475-8e81-eefe8b49cfc7/volumes"
Sep 09 03:26:51.119394 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.119363    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"]
Sep 09 03:26:51.119742 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.119728    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8912a7-9f4f-4475-8e81-eefe8b49cfc7" containerName="test-curl-7601-140566200978144"
Sep 09 03:26:51.119782 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.119744    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8912a7-9f4f-4475-8e81-eefe8b49cfc7" containerName="test-curl-7601-140566200978144"
Sep 09 03:26:51.119861 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.119849    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8912a7-9f4f-4475-8e81-eefe8b49cfc7" containerName="test-curl-7601-140566200978144"
Sep 09 03:26:51.122578 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.122536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.125056 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.125032    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a3a90070-openshift-default-dockercfg-5ngbk\""
Sep 09 03:26:51.138461 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.138436    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"]
Sep 09 03:26:51.248139 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68454801-8b9f-4d0e-9059-f0edac22455b-istiod-ca-cert\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.248139 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-data\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.248328 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-certs\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.248328 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtztv\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-kube-api-access-rtztv\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.248328 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-istio-token\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.248328 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-envoy\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.248328 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-credential-socket\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.248328 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-socket\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.248328 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.248322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68454801-8b9f-4d0e-9059-f0edac22455b-istio-podinfo\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349356 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-istio-token\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349559 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-envoy\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349559 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-credential-socket\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349559 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-socket\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349559 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68454801-8b9f-4d0e-9059-f0edac22455b-istio-podinfo\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349559 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68454801-8b9f-4d0e-9059-f0edac22455b-istiod-ca-cert\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-data\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-certs\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.349795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtztv\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-kube-api-access-rtztv\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.350023 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.349992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-credential-socket\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.350241 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.350216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-socket\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.350342 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.350276    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-data\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.350396 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.350341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-certs\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.350396 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.350361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68454801-8b9f-4d0e-9059-f0edac22455b-istiod-ca-cert\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.352640 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.352616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-envoy\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.352738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.352724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68454801-8b9f-4d0e-9059-f0edac22455b-istio-podinfo\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.357847 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.357801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtztv\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-kube-api-access-rtztv\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.357968 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.357925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-istio-token\") pod \"e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") " pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.433892 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.433820    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:51.719937 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.719911    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"]
Sep 09 03:26:51.725522 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:26:51.725483    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68454801_8b9f_4d0e_9059_f0edac22455b.slice/crio-66b8df285b1661e90b5bd420c00f315a723c1c793e7ebb5c72ce53d4c26b17d8 WatchSource:0}: Error finding container 66b8df285b1661e90b5bd420c00f315a723c1c793e7ebb5c72ce53d4c26b17d8: Status 404 returned error can't find the container with id 66b8df285b1661e90b5bd420c00f315a723c1c793e7ebb5c72ce53d4c26b17d8
Sep 09 03:26:51.727544 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.727512    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:51.727608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.727579    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:51.727647 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.727611    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:26:51.962116 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.962071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" event={"ID":"68454801-8b9f-4d0e-9059-f0edac22455b","Type":"ContainerStarted","Data":"16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c"}
Sep 09 03:26:51.962302 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.962136    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" event={"ID":"68454801-8b9f-4d0e-9059-f0edac22455b","Type":"ContainerStarted","Data":"66b8df285b1661e90b5bd420c00f315a723c1c793e7ebb5c72ce53d4c26b17d8"}
Sep 09 03:26:51.996693 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:51.996595    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" podStartSLOduration=0.996578266 podStartE2EDuration="996.578266ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:51.994616461 +0000 UTC m=+1261.508698611" watchObservedRunningTime="2026-09-09 03:26:51.996578266 +0000 UTC m=+1261.510660416"
Sep 09 03:26:52.434113 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:52.434075    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:52.439292 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:52.439266    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:52.967596 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:52.967567    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:26:52.968529 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:26:52.968509    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:27:05.619655 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.619613    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"]
Sep 09 03:27:05.622706 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.622689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.626390 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.626368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"kube-root-ca.crt\""
Sep 09 03:27:05.626390 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.626391    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"auth-test-model-1a85fa-kserve-self-signed-certs\""
Sep 09 03:27:05.626617 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.626404    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"default-dockercfg-xnl9c\""
Sep 09 03:27:05.626617 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.626505    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"openshift-service-ca.crt\""
Sep 09 03:27:05.647290 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.647265    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"]
Sep 09 03:27:05.692705 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.692665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54c647a-5420-41b9-a12a-f64dfa426afe-tls-certs\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.692705 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.692711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-model-cache\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.692968 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.692761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-tmp-dir\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.692968 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.692850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-home\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.692968 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.692912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-dshm\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.693071 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.693043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq58t\" (UniqueName: \"kubernetes.io/projected/f54c647a-5420-41b9-a12a-f64dfa426afe-kube-api-access-mq58t\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794114 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq58t\" (UniqueName: \"kubernetes.io/projected/f54c647a-5420-41b9-a12a-f64dfa426afe-kube-api-access-mq58t\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54c647a-5420-41b9-a12a-f64dfa426afe-tls-certs\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-model-cache\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-tmp-dir\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-home\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794484 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-dshm\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794584 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794562    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-model-cache\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794629 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-tmp-dir\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.794692 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.794673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-home\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.796613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.796597    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-dshm\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.796993 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.796977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54c647a-5420-41b9-a12a-f64dfa426afe-tls-certs\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.807839 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.807800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq58t\" (UniqueName: \"kubernetes.io/projected/f54c647a-5420-41b9-a12a-f64dfa426afe-kube-api-access-mq58t\") pod \"auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:05.932203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:05.932100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:06.242750 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:06.242665    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"]
Sep 09 03:27:06.245735 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:27:06.245705    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf54c647a_5420_41b9_a12a_f64dfa426afe.slice/crio-5b39d76251237e08bea99d700e8e568b9bacb7c457ef29b2e668012d7e0a0723 WatchSource:0}: Error finding container 5b39d76251237e08bea99d700e8e568b9bacb7c457ef29b2e668012d7e0a0723: Status 404 returned error can't find the container with id 5b39d76251237e08bea99d700e8e568b9bacb7c457ef29b2e668012d7e0a0723
Sep 09 03:27:07.049899 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:07.049860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5" event={"ID":"f54c647a-5420-41b9-a12a-f64dfa426afe","Type":"ContainerStarted","Data":"bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9"}
Sep 09 03:27:07.049899 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:07.049905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5" event={"ID":"f54c647a-5420-41b9-a12a-f64dfa426afe","Type":"ContainerStarted","Data":"5b39d76251237e08bea99d700e8e568b9bacb7c457ef29b2e668012d7e0a0723"}
Sep 09 03:27:07.073691 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:07.073616    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5" podStartSLOduration=2.073577331 podStartE2EDuration="2.073577331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:27:07.072227185 +0000 UTC m=+1276.586309335" watchObservedRunningTime="2026-09-09 03:27:07.073577331 +0000 UTC m=+1276.587659482"
Sep 09 03:27:13.090520 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:13.090429    2583 generic.go:356] "Generic (PLEG): container finished" podID="68454801-8b9f-4d0e-9059-f0edac22455b" containerID="16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c" exitCode=0
Sep 09 03:27:13.091061 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:13.090510    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" event={"ID":"68454801-8b9f-4d0e-9059-f0edac22455b","Type":"ContainerDied","Data":"16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c"}
Sep 09 03:27:13.091126 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:13.091105    2583 scope.go:117] "RemoveContainer" containerID="16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c"
Sep 09 03:27:13.092343 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:13.092315    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:27:13.092411 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:13.092389    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:27:13.092446 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:13.092422    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:27:14.099422 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:14.099388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" event={"ID":"68454801-8b9f-4d0e-9059-f0edac22455b","Type":"ContainerStarted","Data":"8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56"}
Sep 09 03:27:14.434944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:14.434908    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:27:14.435249 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:14.435228    2583 patch_prober.go:28] interesting pod/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 09 03:27:14.435372 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:14.435278    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 09 03:27:15.434711 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:15.434668    2583 patch_prober.go:28] interesting pod/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 09 03:27:15.435139 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:15.434729    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 09 03:27:15.932401 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:15.932367    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:15.932593 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:15.932418    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:15.945658 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:15.945629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:16.123017 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:16.122988    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:27:16.434460 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:16.434423    2583 patch_prober.go:28] interesting pod/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 09 03:27:16.434634 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:16.434487    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 09 03:27:17.434678 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:17.434645    2583 patch_prober.go:28] interesting pod/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 09 03:27:17.435097 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:17.434712    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 09 03:27:18.435069 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:18.435034    2583 patch_prober.go:28] interesting pod/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 09 03:27:18.435458 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:18.435092    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 09 03:27:19.439268 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:19.439229    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:27:19.439732 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:19.439530    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:27:19.440403 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:19.440382    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:27:51.052889 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:51.052860    2583 scope.go:117] "RemoveContainer" containerID="8298b4dd3a10cfecdf73cedc36550e6a41acc04bd9697b98eef8bb224b5b3d66"
Sep 09 03:27:51.061558 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:51.061537    2583 scope.go:117] "RemoveContainer" containerID="7b82455e2e973a820c8d5cd07b0d08809c61afbd27339fcbabbfa219fd4760de"
Sep 09 03:27:56.027780 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.027740    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"]
Sep 09 03:27:56.031184 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.031162    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.040060 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.040031    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"auth-test-model-cd1da6-kserve-self-signed-certs\""
Sep 09 03:27:56.040192 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.040032    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"kube-root-ca.crt\""
Sep 09 03:27:56.040192 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.040032    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"openshift-service-ca.crt\""
Sep 09 03:27:56.040192 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.040098    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"default-dockercfg-vf7q8\""
Sep 09 03:27:56.065248 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.065219    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"]
Sep 09 03:27:56.100557 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.100519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-home\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.100752 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.100583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba52736-6d85-4558-a7aa-87b017a35bd9-tls-certs\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.100752 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.100731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-dshm\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.100878 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.100775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-model-cache\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.100934 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.100869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjqlt\" (UniqueName: \"kubernetes.io/projected/4ba52736-6d85-4558-a7aa-87b017a35bd9-kube-api-access-cjqlt\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.100934 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.100920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-tmp-dir\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.201979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.201939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-home\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.202146 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.202002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba52736-6d85-4558-a7aa-87b017a35bd9-tls-certs\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.202146 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.202077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-dshm\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.202146 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.202106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-model-cache\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.202289 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.202155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjqlt\" (UniqueName: \"kubernetes.io/projected/4ba52736-6d85-4558-a7aa-87b017a35bd9-kube-api-access-cjqlt\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.202289 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.202198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-tmp-dir\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.202387 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.202337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-home\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.202466 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.202444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-model-cache\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.202522 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.202475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-tmp-dir\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.204492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.204469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-dshm\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.204631 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.204614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba52736-6d85-4558-a7aa-87b017a35bd9-tls-certs\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.211472 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.211448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjqlt\" (UniqueName: \"kubernetes.io/projected/4ba52736-6d85-4558-a7aa-87b017a35bd9-kube-api-access-cjqlt\") pod \"auth-test-model-cd1da6-kserve-689cfffb99-dm4pg\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.340194 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.340088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:27:56.617330 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:56.617304    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"]
Sep 09 03:27:56.619560 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:27:56.619531    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ba52736_6d85_4558_a7aa_87b017a35bd9.slice/crio-c620e144205bb0be9a334aa5713ae6940a6a1b9cc7ee41701e58ed5b16e3a608 WatchSource:0}: Error finding container c620e144205bb0be9a334aa5713ae6940a6a1b9cc7ee41701e58ed5b16e3a608: Status 404 returned error can't find the container with id c620e144205bb0be9a334aa5713ae6940a6a1b9cc7ee41701e58ed5b16e3a608
Sep 09 03:27:57.348646 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:57.348603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg" event={"ID":"4ba52736-6d85-4558-a7aa-87b017a35bd9","Type":"ContainerStarted","Data":"54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898"}
Sep 09 03:27:57.349063 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:57.348654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg" event={"ID":"4ba52736-6d85-4558-a7aa-87b017a35bd9","Type":"ContainerStarted","Data":"c620e144205bb0be9a334aa5713ae6940a6a1b9cc7ee41701e58ed5b16e3a608"}
Sep 09 03:27:57.367004 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:27:57.366955    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg" podStartSLOduration=2.3669379360000002 podStartE2EDuration="2.366937936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:27:57.36644363 +0000 UTC m=+1326.880525802" watchObservedRunningTime="2026-09-09 03:27:57.366937936 +0000 UTC m=+1326.881020136"
Sep 09 03:28:06.340329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:06.340289    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:28:06.340862 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:06.340350    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:28:06.352726 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:06.352696    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:28:06.417063 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:06.417031    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:28:24.505996 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.505918    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"]
Sep 09 03:28:24.508330 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.508312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"
Sep 09 03:28:24.510996 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.510973    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:28:24.511145 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.511007    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:28:24.517095 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.516865    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"]
Sep 09 03:28:24.589168 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.589129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ead4b-41a2-4cc7-b837-fab14481ad78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a3a90070-jwthh\" (UID: \"df9ead4b-41a2-4cc7-b837-fab14481ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"
Sep 09 03:28:24.690525 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.690485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ead4b-41a2-4cc7-b837-fab14481ad78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a3a90070-jwthh\" (UID: \"df9ead4b-41a2-4cc7-b837-fab14481ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"
Sep 09 03:28:24.691167 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.691146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ead4b-41a2-4cc7-b837-fab14481ad78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a3a90070-jwthh\" (UID: \"df9ead4b-41a2-4cc7-b837-fab14481ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"
Sep 09 03:28:24.821875 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:24.821777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"
Sep 09 03:28:25.102127 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:25.102105    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"]
Sep 09 03:28:25.104629 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:28:25.104591    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9ead4b_41a2_4cc7_b837_fab14481ad78.slice/crio-cfa7601f087da45c2ff9f999d7146b0b1ea7de04d9653dd25f5213d971165701 WatchSource:0}: Error finding container cfa7601f087da45c2ff9f999d7146b0b1ea7de04d9653dd25f5213d971165701: Status 404 returned error can't find the container with id cfa7601f087da45c2ff9f999d7146b0b1ea7de04d9653dd25f5213d971165701
Sep 09 03:28:25.511416 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:25.511379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh" event={"ID":"df9ead4b-41a2-4cc7-b837-fab14481ad78","Type":"ContainerStarted","Data":"3c18fb9f57faef41ffddf75153b2faff03cfdd1c0649a9c3e790b367ca557e7b"}
Sep 09 03:28:25.511938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:25.511452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh" event={"ID":"df9ead4b-41a2-4cc7-b837-fab14481ad78","Type":"ContainerStarted","Data":"cfa7601f087da45c2ff9f999d7146b0b1ea7de04d9653dd25f5213d971165701"}
Sep 09 03:28:26.519239 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:26.519205    2583 generic.go:356] "Generic (PLEG): container finished" podID="df9ead4b-41a2-4cc7-b837-fab14481ad78" containerID="3c18fb9f57faef41ffddf75153b2faff03cfdd1c0649a9c3e790b367ca557e7b" exitCode=0
Sep 09 03:28:26.519642 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:26.519294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh" event={"ID":"df9ead4b-41a2-4cc7-b837-fab14481ad78","Type":"ContainerDied","Data":"3c18fb9f57faef41ffddf75153b2faff03cfdd1c0649a9c3e790b367ca557e7b"}
Sep 09 03:28:27.666496 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:27.666474    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"
Sep 09 03:28:27.721749 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:27.721718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ead4b-41a2-4cc7-b837-fab14481ad78-maas-api-service-ca\") pod \"df9ead4b-41a2-4cc7-b837-fab14481ad78\" (UID: \"df9ead4b-41a2-4cc7-b837-fab14481ad78\") "
Sep 09 03:28:27.722119 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:27.722093    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9ead4b-41a2-4cc7-b837-fab14481ad78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df9ead4b-41a2-4cc7-b837-fab14481ad78" (UID: "df9ead4b-41a2-4cc7-b837-fab14481ad78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:28:27.823407 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:27.823306    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9ead4b-41a2-4cc7-b837-fab14481ad78-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:28:28.534174 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:28.534148    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"
Sep 09 03:28:28.534354 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:28.534182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh" event={"ID":"df9ead4b-41a2-4cc7-b837-fab14481ad78","Type":"ContainerDied","Data":"cfa7601f087da45c2ff9f999d7146b0b1ea7de04d9653dd25f5213d971165701"}
Sep 09 03:28:28.534354 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:28.534232    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa7601f087da45c2ff9f999d7146b0b1ea7de04d9653dd25f5213d971165701"
Sep 09 03:28:40.461497 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:40.461459    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"]
Sep 09 03:28:40.469548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:40.469516    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3a90070-jwthh"]
Sep 09 03:28:41.008731 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:41.008693    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9ead4b-41a2-4cc7-b837-fab14481ad78" path="/var/lib/kubelet/pods/df9ead4b-41a2-4cc7-b837-fab14481ad78/volumes"
Sep 09 03:28:51.084991 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:51.084961    2583 scope.go:117] "RemoveContainer" containerID="471e8045b3b086bb7c431701dd4e248ae61e993c032c680a7eef2183ad0424d5"
Sep 09 03:28:51.094219 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:28:51.094194    2583 scope.go:117] "RemoveContainer" containerID="be719551669e516f82331c8a2cd699707d3b5c2c9c26593845083a00370fcd64"
Sep 09 03:29:32.022798 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:32.022761    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"]
Sep 09 03:29:32.023352 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:32.022994    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy" containerID="cri-o://8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56" gracePeriod=30
Sep 09 03:29:34.441813 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:34.441773    2583 patch_prober.go:28] interesting pod/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:29:34.442214 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:34.441881    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:29:37.276000 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.275965    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:29:37.326619 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326527    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-envoy\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.326768 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326698    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-certs\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.326815 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326772    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68454801-8b9f-4d0e-9059-f0edac22455b-istiod-ca-cert\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.326815 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326802    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-socket\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.326944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326856    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68454801-8b9f-4d0e-9059-f0edac22455b-istio-podinfo\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.326944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326882    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-data\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.326944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-istio-token\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.326944 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326935    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-credential-socket\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.327093 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.326987    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtztv\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-kube-api-access-rtztv\") pod \"68454801-8b9f-4d0e-9059-f0edac22455b\" (UID: \"68454801-8b9f-4d0e-9059-f0edac22455b\") "
Sep 09 03:29:37.327584 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.327286    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-data" (OuterVolumeSpecName: "istio-data") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:37.327584 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.327355    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:37.327584 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.327398    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:37.327584 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.327535    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.327584 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.327555    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.327584 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.327568    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.327584 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.327537    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68454801-8b9f-4d0e-9059-f0edac22455b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:29:37.327992 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.327593    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:37.329481 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.329458    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:29:37.329603 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.329481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-kube-api-access-rtztv" (OuterVolumeSpecName: "kube-api-access-rtztv") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "kube-api-access-rtztv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:29:37.329764 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.329728    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68454801-8b9f-4d0e-9059-f0edac22455b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:29:37.329882 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.329770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-istio-token" (OuterVolumeSpecName: "istio-token") pod "68454801-8b9f-4d0e-9059-f0edac22455b" (UID: "68454801-8b9f-4d0e-9059-f0edac22455b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:29:37.428260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.428225    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.428260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.428254    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68454801-8b9f-4d0e-9059-f0edac22455b-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.428260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.428263    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68454801-8b9f-4d0e-9059-f0edac22455b-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.428489 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.428271    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.428489 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.428281    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68454801-8b9f-4d0e-9059-f0edac22455b-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.428489 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.428290    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtztv\" (UniqueName: \"kubernetes.io/projected/68454801-8b9f-4d0e-9059-f0edac22455b-kube-api-access-rtztv\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:29:37.918999 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.918963    2583 generic.go:356] "Generic (PLEG): container finished" podID="68454801-8b9f-4d0e-9059-f0edac22455b" containerID="8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56" exitCode=0
Sep 09 03:29:37.919203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.919014    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"
Sep 09 03:29:37.919203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.919053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" event={"ID":"68454801-8b9f-4d0e-9059-f0edac22455b","Type":"ContainerDied","Data":"8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56"}
Sep 09 03:29:37.919203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.919114    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl" event={"ID":"68454801-8b9f-4d0e-9059-f0edac22455b","Type":"ContainerDied","Data":"66b8df285b1661e90b5bd420c00f315a723c1c793e7ebb5c72ce53d4c26b17d8"}
Sep 09 03:29:37.919203 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.919144    2583 scope.go:117] "RemoveContainer" containerID="8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56"
Sep 09 03:29:37.928985 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.928969    2583 scope.go:117] "RemoveContainer" containerID="16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c"
Sep 09 03:29:37.937179 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.937162    2583 scope.go:117] "RemoveContainer" containerID="8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56"
Sep 09 03:29:37.937459 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:29:37.937441    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56\": container with ID starting with 8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56 not found: ID does not exist" containerID="8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56"
Sep 09 03:29:37.937508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.937468    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56"} err="failed to get container status \"8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56\": rpc error: code = NotFound desc = could not find container \"8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56\": container with ID starting with 8ed9d9bfe1774286c11badd3b9c971ee00d2ea98184917204f6ff59890494e56 not found: ID does not exist"
Sep 09 03:29:37.937508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.937485    2583 scope.go:117] "RemoveContainer" containerID="16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c"
Sep 09 03:29:37.937727 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:29:37.937709    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c\": container with ID starting with 16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c not found: ID does not exist" containerID="16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c"
Sep 09 03:29:37.937782 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.937737    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c"} err="failed to get container status \"16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c\": rpc error: code = NotFound desc = could not find container \"16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c\": container with ID starting with 16dc1c730092200cd499062581f3ba9b0fde3241d267bafce588650e079d9a1c not found: ID does not exist"
Sep 09 03:29:37.942578 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.942557    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"]
Sep 09 03:29:37.949943 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:37.949924    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a3a90070-openshift-default-5cc9c7c9b8-l5kwl"]
Sep 09 03:29:39.007618 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:29:39.007585    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" path="/var/lib/kubelet/pods/68454801-8b9f-4d0e-9059-f0edac22455b/volumes"
Sep 09 03:30:00.156072 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156014    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"]
Sep 09 03:30:00.156508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156493    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy"
Sep 09 03:30:00.156551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156510    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy"
Sep 09 03:30:00.156551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156525    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy"
Sep 09 03:30:00.156551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156531    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy"
Sep 09 03:30:00.156551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156538    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9ead4b-41a2-4cc7-b837-fab14481ad78" containerName="revoke-keys"
Sep 09 03:30:00.156551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156543    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9ead4b-41a2-4cc7-b837-fab14481ad78" containerName="revoke-keys"
Sep 09 03:30:00.156702 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156613    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy"
Sep 09 03:30:00.156702 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156623    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="68454801-8b9f-4d0e-9059-f0edac22455b" containerName="istio-proxy"
Sep 09 03:30:00.156702 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.156629    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9ead4b-41a2-4cc7-b837-fab14481ad78" containerName="revoke-keys"
Sep 09 03:30:00.161088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.161066    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"]
Sep 09 03:30:00.161234 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.161188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"
Sep 09 03:30:00.164350 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.164331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"
Sep 09 03:30:00.164474 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.164401    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:30:00.172806 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.172784    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"
Sep 09 03:30:00.181163 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.181043    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"]
Sep 09 03:30:00.181729 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.181711    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"
Sep 09 03:30:00.183564 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.183540    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"]
Sep 09 03:30:00.459071 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.459045    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"]
Sep 09 03:30:00.460863 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:30:00.460818    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797d5ae8_3351_481c_aa75_075ebac647ae.slice/crio-f0208f518aa013d28e609c79364b7d1083a93a421087aac03ff6de579f99a380 WatchSource:0}: Error finding container f0208f518aa013d28e609c79364b7d1083a93a421087aac03ff6de579f99a380: Status 404 returned error can't find the container with id f0208f518aa013d28e609c79364b7d1083a93a421087aac03ff6de579f99a380
Sep 09 03:30:00.475000 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:00.474977    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"]
Sep 09 03:30:00.477663 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:30:00.477625    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ca3e47_2200_40da_b4f0_96e34674711f.slice/crio-03d8ee05ef7d40f4187ff7ae7785e61862e54b4963b1bdc84128b1ff3844e36e WatchSource:0}: Error finding container 03d8ee05ef7d40f4187ff7ae7785e61862e54b4963b1bdc84128b1ff3844e36e: Status 404 returned error can't find the container with id 03d8ee05ef7d40f4187ff7ae7785e61862e54b4963b1bdc84128b1ff3844e36e
Sep 09 03:30:01.052678 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:01.052581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld" event={"ID":"797d5ae8-3351-481c-aa75-075ebac647ae","Type":"ContainerStarted","Data":"d703f7bd1c16ef2fcf0f88992d1d904cb389784151a7dc876991c87829862aaf"}
Sep 09 03:30:01.052678 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:01.052647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld" event={"ID":"797d5ae8-3351-481c-aa75-075ebac647ae","Type":"ContainerStarted","Data":"f0208f518aa013d28e609c79364b7d1083a93a421087aac03ff6de579f99a380"}
Sep 09 03:30:01.054089 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:01.054069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps" event={"ID":"10ca3e47-2200-40da-b4f0-96e34674711f","Type":"ContainerStarted","Data":"440fdf02bffa1039d8370cccbd93aa5465f87440e78eca01b6b99350c91392ae"}
Sep 09 03:30:01.054164 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:01.054107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps" event={"ID":"10ca3e47-2200-40da-b4f0-96e34674711f","Type":"ContainerStarted","Data":"03d8ee05ef7d40f4187ff7ae7785e61862e54b4963b1bdc84128b1ff3844e36e"}
Sep 09 03:30:01.070367 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:01.070311    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld" podStartSLOduration=1.070296343 podStartE2EDuration="1.070296343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:01.069294798 +0000 UTC m=+1450.583376951" watchObservedRunningTime="2026-09-09 03:30:01.070296343 +0000 UTC m=+1450.584378490"
Sep 09 03:30:01.084355 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:01.084312    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps" podStartSLOduration=1.084298285 podStartE2EDuration="1.084298285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:01.083336956 +0000 UTC m=+1450.597419117" watchObservedRunningTime="2026-09-09 03:30:01.084298285 +0000 UTC m=+1450.598380486"
Sep 09 03:30:02.061846 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:02.061796    2583 generic.go:356] "Generic (PLEG): container finished" podID="10ca3e47-2200-40da-b4f0-96e34674711f" containerID="440fdf02bffa1039d8370cccbd93aa5465f87440e78eca01b6b99350c91392ae" exitCode=0
Sep 09 03:30:02.062326 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:02.061888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps" event={"ID":"10ca3e47-2200-40da-b4f0-96e34674711f","Type":"ContainerDied","Data":"440fdf02bffa1039d8370cccbd93aa5465f87440e78eca01b6b99350c91392ae"}
Sep 09 03:30:02.063489 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:02.063467    2583 generic.go:356] "Generic (PLEG): container finished" podID="797d5ae8-3351-481c-aa75-075ebac647ae" containerID="d703f7bd1c16ef2fcf0f88992d1d904cb389784151a7dc876991c87829862aaf" exitCode=0
Sep 09 03:30:02.063565 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:02.063547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld" event={"ID":"797d5ae8-3351-481c-aa75-075ebac647ae","Type":"ContainerDied","Data":"d703f7bd1c16ef2fcf0f88992d1d904cb389784151a7dc876991c87829862aaf"}
Sep 09 03:30:03.211854 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:03.211817    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"
Sep 09 03:30:03.214795 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:03.214774    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"
Sep 09 03:30:04.075995 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:04.075961    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"
Sep 09 03:30:04.075995 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:04.075991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps" event={"ID":"10ca3e47-2200-40da-b4f0-96e34674711f","Type":"ContainerDied","Data":"03d8ee05ef7d40f4187ff7ae7785e61862e54b4963b1bdc84128b1ff3844e36e"}
Sep 09 03:30:04.076274 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:04.076053    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03d8ee05ef7d40f4187ff7ae7785e61862e54b4963b1bdc84128b1ff3844e36e"
Sep 09 03:30:04.077403 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:04.077374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld" event={"ID":"797d5ae8-3351-481c-aa75-075ebac647ae","Type":"ContainerDied","Data":"f0208f518aa013d28e609c79364b7d1083a93a421087aac03ff6de579f99a380"}
Sep 09 03:30:04.077535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:04.077408    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0208f518aa013d28e609c79364b7d1083a93a421087aac03ff6de579f99a380"
Sep 09 03:30:04.077535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:04.077380    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"
Sep 09 03:30:25.810530 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:25.810495    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"]
Sep 09 03:30:27.200955 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:27.200925    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"]
Sep 09 03:30:39.706668 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.706633    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"]
Sep 09 03:30:39.707278 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.707256    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ca3e47-2200-40da-b4f0-96e34674711f" containerName="cleanup"
Sep 09 03:30:39.707327 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.707284    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ca3e47-2200-40da-b4f0-96e34674711f" containerName="cleanup"
Sep 09 03:30:39.707327 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.707296    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797d5ae8-3351-481c-aa75-075ebac647ae" containerName="cleanup"
Sep 09 03:30:39.707327 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.707305    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="797d5ae8-3351-481c-aa75-075ebac647ae" containerName="cleanup"
Sep 09 03:30:39.707448 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.707436    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="797d5ae8-3351-481c-aa75-075ebac647ae" containerName="cleanup"
Sep 09 03:30:39.707483 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.707451    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ca3e47-2200-40da-b4f0-96e34674711f" containerName="cleanup"
Sep 09 03:30:39.711260 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.711237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.714596 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.714576    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1a85fa\"/\"sub-test-model-1a85fa-kserve-self-signed-certs\""
Sep 09 03:30:39.721225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.721195    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"]
Sep 09 03:30:39.810879 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.810856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-dshm\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.810984 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.810889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tls-certs\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.810984 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.810921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tmp-dir\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.810984 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.810940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-model-cache\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.810984 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.810962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-home\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.811123 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.811070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbfn\" (UniqueName: \"kubernetes.io/projected/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-kube-api-access-vgbfn\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912039 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.911987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tmp-dir\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912039 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.912037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-model-cache\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912242 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.912065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-home\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912242 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.912119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbfn\" (UniqueName: \"kubernetes.io/projected/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-kube-api-access-vgbfn\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912242 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.912165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-dshm\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912242 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.912182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tls-certs\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912466 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.912443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tmp-dir\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.912473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-home\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.912551 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.912542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-model-cache\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.914566 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.914546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-dshm\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.914724 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.914707    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tls-certs\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:39.921256 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:39.921238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbfn\" (UniqueName: \"kubernetes.io/projected/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-kube-api-access-vgbfn\") pod \"sub-test-model-1a85fa-kserve-866dbc54c5-lprzc\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") " pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:40.023613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:40.023563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:40.300549 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:40.300526    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"]
Sep 09 03:30:40.303012 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:30:40.302986    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2539708_c61a_4c6d_a2e7_8a6bab96fbaa.slice/crio-dd6e994034e1549bce4dde900c1f88dc3b220ae6ebfb7407ba286df437bfd4e9 WatchSource:0}: Error finding container dd6e994034e1549bce4dde900c1f88dc3b220ae6ebfb7407ba286df437bfd4e9: Status 404 returned error can't find the container with id dd6e994034e1549bce4dde900c1f88dc3b220ae6ebfb7407ba286df437bfd4e9
Sep 09 03:30:40.830556 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:40.830492    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5" podUID="f54c647a-5420-41b9-a12a-f64dfa426afe" containerName="main" containerID="cri-o://bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9" gracePeriod=45
Sep 09 03:30:41.075713 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.075674    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:30:41.224776 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.224748    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54c647a-5420-41b9-a12a-f64dfa426afe-tls-certs\") pod \"f54c647a-5420-41b9-a12a-f64dfa426afe\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") "
Sep 09 03:30:41.224976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.224866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-model-cache\") pod \"f54c647a-5420-41b9-a12a-f64dfa426afe\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") "
Sep 09 03:30:41.224976 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.224961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-home\") pod \"f54c647a-5420-41b9-a12a-f64dfa426afe\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") "
Sep 09 03:30:41.225092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.224998    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-dshm\") pod \"f54c647a-5420-41b9-a12a-f64dfa426afe\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") "
Sep 09 03:30:41.225092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.225025    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-tmp-dir\") pod \"f54c647a-5420-41b9-a12a-f64dfa426afe\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") "
Sep 09 03:30:41.225092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.225052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq58t\" (UniqueName: \"kubernetes.io/projected/f54c647a-5420-41b9-a12a-f64dfa426afe-kube-api-access-mq58t\") pod \"f54c647a-5420-41b9-a12a-f64dfa426afe\" (UID: \"f54c647a-5420-41b9-a12a-f64dfa426afe\") "
Sep 09 03:30:41.225261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.225100    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-model-cache" (OuterVolumeSpecName: "model-cache") pod "f54c647a-5420-41b9-a12a-f64dfa426afe" (UID: "f54c647a-5420-41b9-a12a-f64dfa426afe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:41.225261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.225159    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-home" (OuterVolumeSpecName: "home") pod "f54c647a-5420-41b9-a12a-f64dfa426afe" (UID: "f54c647a-5420-41b9-a12a-f64dfa426afe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:41.225376 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.225313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f54c647a-5420-41b9-a12a-f64dfa426afe" (UID: "f54c647a-5420-41b9-a12a-f64dfa426afe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:41.225548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.225524    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:41.225670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.225552    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:41.225670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.225564    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:41.227283 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.227254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c647a-5420-41b9-a12a-f64dfa426afe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f54c647a-5420-41b9-a12a-f64dfa426afe" (UID: "f54c647a-5420-41b9-a12a-f64dfa426afe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:30:41.227402 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.227325    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54c647a-5420-41b9-a12a-f64dfa426afe-kube-api-access-mq58t" (OuterVolumeSpecName: "kube-api-access-mq58t") pod "f54c647a-5420-41b9-a12a-f64dfa426afe" (UID: "f54c647a-5420-41b9-a12a-f64dfa426afe"). InnerVolumeSpecName "kube-api-access-mq58t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:41.227609 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.227593    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-dshm" (OuterVolumeSpecName: "dshm") pod "f54c647a-5420-41b9-a12a-f64dfa426afe" (UID: "f54c647a-5420-41b9-a12a-f64dfa426afe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:41.272644 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.272608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc" event={"ID":"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa","Type":"ContainerStarted","Data":"fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc"}
Sep 09 03:30:41.272790 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.272657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc" event={"ID":"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa","Type":"ContainerStarted","Data":"dd6e994034e1549bce4dde900c1f88dc3b220ae6ebfb7407ba286df437bfd4e9"}
Sep 09 03:30:41.274170 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.274144    2583 generic.go:356] "Generic (PLEG): container finished" podID="f54c647a-5420-41b9-a12a-f64dfa426afe" containerID="bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9" exitCode=0
Sep 09 03:30:41.274296 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.274200    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"
Sep 09 03:30:41.274296 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.274229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5" event={"ID":"f54c647a-5420-41b9-a12a-f64dfa426afe","Type":"ContainerDied","Data":"bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9"}
Sep 09 03:30:41.274296 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.274273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5" event={"ID":"f54c647a-5420-41b9-a12a-f64dfa426afe","Type":"ContainerDied","Data":"5b39d76251237e08bea99d700e8e568b9bacb7c457ef29b2e668012d7e0a0723"}
Sep 09 03:30:41.274296 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.274290    2583 scope.go:117] "RemoveContainer" containerID="bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9"
Sep 09 03:30:41.284384 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.284368    2583 scope.go:117] "RemoveContainer" containerID="bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9"
Sep 09 03:30:41.284670 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:30:41.284654    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9\": container with ID starting with bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9 not found: ID does not exist" containerID="bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9"
Sep 09 03:30:41.284715 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.284676    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9"} err="failed to get container status \"bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9\": rpc error: code = NotFound desc = could not find container \"bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9\": container with ID starting with bfa6e29233c6da4c4f8f6d6712bfb6202f63a47fba4ec7cc9f107b533eef71d9 not found: ID does not exist"
Sep 09 03:30:41.291693 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.291652    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc" podStartSLOduration=2.291637723 podStartE2EDuration="2.291637723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:41.290257194 +0000 UTC m=+1490.804339368" watchObservedRunningTime="2026-09-09 03:30:41.291637723 +0000 UTC m=+1490.805719872"
Sep 09 03:30:41.308319 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.308278    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"]
Sep 09 03:30:41.310538 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.310519    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/auth-test-model-1a85fa-kserve-54c49f4ff-f8tb5"]
Sep 09 03:30:41.326113 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.326097    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54c647a-5420-41b9-a12a-f64dfa426afe-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:41.326212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.326115    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq58t\" (UniqueName: \"kubernetes.io/projected/f54c647a-5420-41b9-a12a-f64dfa426afe-kube-api-access-mq58t\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:41.326212 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:41.326128    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54c647a-5420-41b9-a12a-f64dfa426afe-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:42.220959 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.220907    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg" podUID="4ba52736-6d85-4558-a7aa-87b017a35bd9" containerName="main" containerID="cri-o://54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898" gracePeriod=45
Sep 09 03:30:42.463570 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.463546    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:30:42.537717 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.537647    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-tmp-dir\") pod \"4ba52736-6d85-4558-a7aa-87b017a35bd9\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") "
Sep 09 03:30:42.537891 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.537732    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-home\") pod \"4ba52736-6d85-4558-a7aa-87b017a35bd9\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") "
Sep 09 03:30:42.537891 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.537794    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-model-cache\") pod \"4ba52736-6d85-4558-a7aa-87b017a35bd9\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") "
Sep 09 03:30:42.537891 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.537863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba52736-6d85-4558-a7aa-87b017a35bd9-tls-certs\") pod \"4ba52736-6d85-4558-a7aa-87b017a35bd9\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") "
Sep 09 03:30:42.537891 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.537887    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-dshm\") pod \"4ba52736-6d85-4558-a7aa-87b017a35bd9\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") "
Sep 09 03:30:42.538160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.537950    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjqlt\" (UniqueName: \"kubernetes.io/projected/4ba52736-6d85-4558-a7aa-87b017a35bd9-kube-api-access-cjqlt\") pod \"4ba52736-6d85-4558-a7aa-87b017a35bd9\" (UID: \"4ba52736-6d85-4558-a7aa-87b017a35bd9\") "
Sep 09 03:30:42.538160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.537972    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ba52736-6d85-4558-a7aa-87b017a35bd9" (UID: "4ba52736-6d85-4558-a7aa-87b017a35bd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:42.538160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.538032    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-home" (OuterVolumeSpecName: "home") pod "4ba52736-6d85-4558-a7aa-87b017a35bd9" (UID: "4ba52736-6d85-4558-a7aa-87b017a35bd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:42.538160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.538046    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ba52736-6d85-4558-a7aa-87b017a35bd9" (UID: "4ba52736-6d85-4558-a7aa-87b017a35bd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:42.538330 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.538290    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:42.538330 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.538304    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:42.538330 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.538312    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:42.540206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.540175    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-dshm" (OuterVolumeSpecName: "dshm") pod "4ba52736-6d85-4558-a7aa-87b017a35bd9" (UID: "4ba52736-6d85-4558-a7aa-87b017a35bd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:42.540206 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.540191    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba52736-6d85-4558-a7aa-87b017a35bd9-kube-api-access-cjqlt" (OuterVolumeSpecName: "kube-api-access-cjqlt") pod "4ba52736-6d85-4558-a7aa-87b017a35bd9" (UID: "4ba52736-6d85-4558-a7aa-87b017a35bd9"). InnerVolumeSpecName "kube-api-access-cjqlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:42.540337 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.540226    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba52736-6d85-4558-a7aa-87b017a35bd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ba52736-6d85-4558-a7aa-87b017a35bd9" (UID: "4ba52736-6d85-4558-a7aa-87b017a35bd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:30:42.639642 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.639617    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba52736-6d85-4558-a7aa-87b017a35bd9-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:42.639642 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.639638    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba52736-6d85-4558-a7aa-87b017a35bd9-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:42.639777 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:42.639647    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjqlt\" (UniqueName: \"kubernetes.io/projected/4ba52736-6d85-4558-a7aa-87b017a35bd9-kube-api-access-cjqlt\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:30:43.008293 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.008263    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f54c647a-5420-41b9-a12a-f64dfa426afe" path="/var/lib/kubelet/pods/f54c647a-5420-41b9-a12a-f64dfa426afe/volumes"
Sep 09 03:30:43.289311 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.289229    2583 generic.go:356] "Generic (PLEG): container finished" podID="4ba52736-6d85-4558-a7aa-87b017a35bd9" containerID="54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898" exitCode=0
Sep 09 03:30:43.289311 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.289269    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"
Sep 09 03:30:43.289741 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.289316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg" event={"ID":"4ba52736-6d85-4558-a7aa-87b017a35bd9","Type":"ContainerDied","Data":"54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898"}
Sep 09 03:30:43.289741 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.289358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg" event={"ID":"4ba52736-6d85-4558-a7aa-87b017a35bd9","Type":"ContainerDied","Data":"c620e144205bb0be9a334aa5713ae6940a6a1b9cc7ee41701e58ed5b16e3a608"}
Sep 09 03:30:43.289741 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.289377    2583 scope.go:117] "RemoveContainer" containerID="54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898"
Sep 09 03:30:43.298038 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.298020    2583 scope.go:117] "RemoveContainer" containerID="54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898"
Sep 09 03:30:43.298322 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:30:43.298303    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898\": container with ID starting with 54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898 not found: ID does not exist" containerID="54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898"
Sep 09 03:30:43.298376 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.298331    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898"} err="failed to get container status \"54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898\": rpc error: code = NotFound desc = could not find container \"54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898\": container with ID starting with 54b12c70e65204f5aabc077b2b8de82e6d2dc4d9651b88662ec9ad336f58d898 not found: ID does not exist"
Sep 09 03:30:43.306743 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.306721    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"]
Sep 09 03:30:43.310201 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:43.310182    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/auth-test-model-cd1da6-kserve-689cfffb99-dm4pg"]
Sep 09 03:30:45.007332 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:45.007299    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba52736-6d85-4558-a7aa-87b017a35bd9" path="/var/lib/kubelet/pods/4ba52736-6d85-4558-a7aa-87b017a35bd9/volumes"
Sep 09 03:30:50.023905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:50.023870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:50.024323 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:50.023920    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:50.036297 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:50.036267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:50.338262 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:50.338190    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:30:51.019221 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:51.019195    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:30:51.021323 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:51.021284    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:30:51.025333 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:51.025315    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:30:51.027492 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:51.027473    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:30:51.132718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:51.132694    2583 scope.go:117] "RemoveContainer" containerID="70905f74a0cb25fd495b3bea1c964f5a5a7a450f45fea64a5a6852ea91dfbc2a"
Sep 09 03:30:56.001860 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.001795    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"]
Sep 09 03:30:56.002329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.002185    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c647a-5420-41b9-a12a-f64dfa426afe" containerName="main"
Sep 09 03:30:56.002329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.002198    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c647a-5420-41b9-a12a-f64dfa426afe" containerName="main"
Sep 09 03:30:56.002329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.002225    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba52736-6d85-4558-a7aa-87b017a35bd9" containerName="main"
Sep 09 03:30:56.002329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.002230    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba52736-6d85-4558-a7aa-87b017a35bd9" containerName="main"
Sep 09 03:30:56.002329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.002280    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba52736-6d85-4558-a7aa-87b017a35bd9" containerName="main"
Sep 09 03:30:56.002329 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.002288    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c647a-5420-41b9-a12a-f64dfa426afe" containerName="main"
Sep 09 03:30:56.006864 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.006822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.009353 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.009335    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-811c96-openshift-default-dockercfg-jvl9r\""
Sep 09 03:30:56.019538 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.019514    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"]
Sep 09 03:30:56.162407 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-data\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.162579 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162423    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-envoy\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.162579 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-credential-socket\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.162579 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b9912-cd9a-4563-873e-9268aa144f1e-istiod-ca-cert\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.162579 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-certs\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.162714 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-podinfo\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.162714 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94x98\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-kube-api-access-94x98\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.162714 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-socket\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.162805 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.162719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-token\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264226 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-envoy\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264226 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-credential-socket\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264226 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b9912-cd9a-4563-873e-9268aa144f1e-istiod-ca-cert\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-certs\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-podinfo\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94x98\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-kube-api-access-94x98\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264518 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-socket\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264720 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-token\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264720 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-data\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264720 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-credential-socket\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.264918 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.264742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-certs\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.265066 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.265039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-socket\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.265148 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.265075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-data\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.265148 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.265101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b9912-cd9a-4563-873e-9268aa144f1e-istiod-ca-cert\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.266712 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.266684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-envoy\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.267029 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.267011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-podinfo\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.275402 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.275375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-token\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.275744 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.275728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94x98\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-kube-api-access-94x98\") pod \"e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") " pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.320344 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.320309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:56.604111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.603997    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"]
Sep 09 03:30:56.610818 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:30:56.610735    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0b9912_cd9a_4563_873e_9268aa144f1e.slice/crio-3e4556b86dd10465420431636b9f89abd58bf89842a3931938c4a197e5325a18 WatchSource:0}: Error finding container 3e4556b86dd10465420431636b9f89abd58bf89842a3931938c4a197e5325a18: Status 404 returned error can't find the container with id 3e4556b86dd10465420431636b9f89abd58bf89842a3931938c4a197e5325a18
Sep 09 03:30:56.613313 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.613274    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:30:56.613416 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.613345    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:30:56.613416 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:56.613389    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:30:57.370456 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:57.370421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" event={"ID":"9b0b9912-cd9a-4563-873e-9268aa144f1e","Type":"ContainerStarted","Data":"39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d"}
Sep 09 03:30:57.370875 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:57.370475    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" event={"ID":"9b0b9912-cd9a-4563-873e-9268aa144f1e","Type":"ContainerStarted","Data":"3e4556b86dd10465420431636b9f89abd58bf89842a3931938c4a197e5325a18"}
Sep 09 03:30:57.393503 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:57.393456    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" podStartSLOduration=2.393436346 podStartE2EDuration="2.393436346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:57.392369714 +0000 UTC m=+1506.906451886" watchObservedRunningTime="2026-09-09 03:30:57.393436346 +0000 UTC m=+1506.907518494"
Sep 09 03:30:58.321517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:58.321482    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:58.326543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:58.326516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:58.376340 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:58.376311    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:30:58.377491 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:30:58.377473    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:31:01.348390 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.348357    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"]
Sep 09 03:31:01.351756 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.351740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:01.354338 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.354314    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-811c96\""
Sep 09 03:31:01.354465 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.354341    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-811c96-dockercfg-zbsc6\""
Sep 09 03:31:01.360925 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.360901    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"]
Sep 09 03:31:01.411165 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.411124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w49c9\" (UniqueName: \"kubernetes.io/projected/193e7204-e729-44a0-aace-d308f2a10d36-kube-api-access-w49c9\") pod \"payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2\" (UID: \"193e7204-e729-44a0-aace-d308f2a10d36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:01.411370 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.411268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e7204-e729-44a0-aace-d308f2a10d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2\" (UID: \"193e7204-e729-44a0-aace-d308f2a10d36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:01.512601 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.512542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w49c9\" (UniqueName: \"kubernetes.io/projected/193e7204-e729-44a0-aace-d308f2a10d36-kube-api-access-w49c9\") pod \"payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2\" (UID: \"193e7204-e729-44a0-aace-d308f2a10d36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:01.512778 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.512653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e7204-e729-44a0-aace-d308f2a10d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2\" (UID: \"193e7204-e729-44a0-aace-d308f2a10d36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:01.513369 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.513347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e7204-e729-44a0-aace-d308f2a10d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2\" (UID: \"193e7204-e729-44a0-aace-d308f2a10d36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:01.520707 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.520679    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w49c9\" (UniqueName: \"kubernetes.io/projected/193e7204-e729-44a0-aace-d308f2a10d36-kube-api-access-w49c9\") pod \"payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2\" (UID: \"193e7204-e729-44a0-aace-d308f2a10d36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:01.664090 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.664048    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:01.965944 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:31:01.965910    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193e7204_e729_44a0_aace_d308f2a10d36.slice/crio-7a6e2cbcc0af3ec0c5b959f6ad47413c2ec58aef598c2b41cc69515079bb1d0d WatchSource:0}: Error finding container 7a6e2cbcc0af3ec0c5b959f6ad47413c2ec58aef598c2b41cc69515079bb1d0d: Status 404 returned error can't find the container with id 7a6e2cbcc0af3ec0c5b959f6ad47413c2ec58aef598c2b41cc69515079bb1d0d
Sep 09 03:31:01.974798 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:01.970656    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"]
Sep 09 03:31:02.403674 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:02.403640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2" event={"ID":"193e7204-e729-44a0-aace-d308f2a10d36","Type":"ContainerStarted","Data":"7a6e2cbcc0af3ec0c5b959f6ad47413c2ec58aef598c2b41cc69515079bb1d0d"}
Sep 09 03:31:04.415048 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:04.415010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2" event={"ID":"193e7204-e729-44a0-aace-d308f2a10d36","Type":"ContainerStarted","Data":"d5b6b6c6f5866bd1cf52dc25d6ddd7fc10e4724a63e641875f789836811bf4ba"}
Sep 09 03:31:04.415397 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:04.415117    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:04.433340 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:04.433296    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2" podStartSLOduration=1.6499557500000002 podStartE2EDuration="3.433281474s" totalImagesPullingTime="1.783325724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:04.432083593 +0000 UTC m=+1513.946165774" watchObservedRunningTime="2026-09-09 03:31:04.433281474 +0000 UTC m=+1513.947363622"
Sep 09 03:31:10.901257 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:10.901217    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"]
Sep 09 03:31:10.904609 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:10.904584    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:10.909032 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:10.909003    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"default-dockercfg-vf7q8\""
Sep 09 03:31:10.909173 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:10.909068    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"sub-test-model-cd1da6-kserve-self-signed-certs\""
Sep 09 03:31:10.909242 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:10.909169    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"openshift-service-ca.crt\""
Sep 09 03:31:10.909322 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:10.909306    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"kube-root-ca.crt\""
Sep 09 03:31:10.917713 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:10.917689    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"]
Sep 09 03:31:11.016583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.016547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb6582f-431a-49a4-bfc2-30052d329ef7-tls-certs\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.016784 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.016616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4hq\" (UniqueName: \"kubernetes.io/projected/ccb6582f-431a-49a4-bfc2-30052d329ef7-kube-api-access-5b4hq\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.016784 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.016641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-home\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.016784 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.016661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-tmp-dir\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.016784 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.016720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-model-cache\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.016784 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.016762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-dshm\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118043 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-dshm\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb6582f-431a-49a4-bfc2-30052d329ef7-tls-certs\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4hq\" (UniqueName: \"kubernetes.io/projected/ccb6582f-431a-49a4-bfc2-30052d329ef7-kube-api-access-5b4hq\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118144    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-home\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-tmp-dir\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118344 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-model-cache\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118508 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-home\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118651 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118564    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-tmp-dir\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.118651 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.118605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-model-cache\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.120481 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.120459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-dshm\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.120748 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.120730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb6582f-431a-49a4-bfc2-30052d329ef7-tls-certs\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.126364 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.126342    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4hq\" (UniqueName: \"kubernetes.io/projected/ccb6582f-431a-49a4-bfc2-30052d329ef7-kube-api-access-5b4hq\") pod \"sub-test-model-cd1da6-kserve-7d8644f59c-qblr9\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.214534 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.214431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:11.512885 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:11.512853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"]
Sep 09 03:31:11.518275 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:31:11.518241    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccb6582f_431a_49a4_bfc2_30052d329ef7.slice/crio-b05cbee9531aa12517e2c537792aabbc6a84b2d8ac334a8d2f5da08001b31502 WatchSource:0}: Error finding container b05cbee9531aa12517e2c537792aabbc6a84b2d8ac334a8d2f5da08001b31502: Status 404 returned error can't find the container with id b05cbee9531aa12517e2c537792aabbc6a84b2d8ac334a8d2f5da08001b31502
Sep 09 03:31:12.463186 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:12.463094    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9" event={"ID":"ccb6582f-431a-49a4-bfc2-30052d329ef7","Type":"ContainerStarted","Data":"1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d"}
Sep 09 03:31:12.463186 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:12.463141    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9" event={"ID":"ccb6582f-431a-49a4-bfc2-30052d329ef7","Type":"ContainerStarted","Data":"b05cbee9531aa12517e2c537792aabbc6a84b2d8ac334a8d2f5da08001b31502"}
Sep 09 03:31:12.487332 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:12.487247    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9" podStartSLOduration=2.487224514 podStartE2EDuration="2.487224514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:12.486318738 +0000 UTC m=+1522.000400919" watchObservedRunningTime="2026-09-09 03:31:12.487224514 +0000 UTC m=+1522.001306662"
Sep 09 03:31:14.418997 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:14.418959    2583 patch_prober.go:28] interesting pod/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 03:31:14.419390 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:14.419023    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 03:31:15.423071 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:15.422996    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:31:15.488447 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:15.488410    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerID="39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d" exitCode=0
Sep 09 03:31:15.488645 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:15.488472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" event={"ID":"9b0b9912-cd9a-4563-873e-9268aa144f1e","Type":"ContainerDied","Data":"39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d"}
Sep 09 03:31:15.489008 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:15.488984    2583 scope.go:117] "RemoveContainer" containerID="39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d"
Sep 09 03:31:15.490438 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:15.490402    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:31:15.490547 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:15.490496    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:31:15.490547 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:15.490537    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:31:16.498114 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:16.498074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" event={"ID":"9b0b9912-cd9a-4563-873e-9268aa144f1e","Type":"ContainerStarted","Data":"bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91"}
Sep 09 03:31:17.320822 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:17.320787    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:31:17.321040 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:17.321019    2583 patch_prober.go:28] interesting pod/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 03:31:17.321134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:17.321076    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 03:31:18.320856 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:18.320796    2583 patch_prober.go:28] interesting pod/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 03:31:18.321281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:18.320892    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 03:31:19.320886 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:19.320821    2583 patch_prober.go:28] interesting pod/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 03:31:19.321370 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:19.320915    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 03:31:20.321209 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:20.321174    2583 patch_prober.go:28] interesting pod/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 03:31:20.321600 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:20.321241    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 03:31:21.214990 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:21.214951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:21.214990 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:21.215005    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:21.227603 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:21.227568    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:21.321200 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:21.321162    2583 patch_prober.go:28] interesting pod/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 03:31:21.321371 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:21.321230    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 03:31:21.536487 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:21.536401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:31:22.325137 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:22.325086    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:31:22.325583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:22.325486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:31:22.326318 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:22.326297    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:31:27.226945 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.226906    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"]
Sep 09 03:31:27.229700 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.229682    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"
Sep 09 03:31:27.233616 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.233594    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvfxb\""
Sep 09 03:31:27.233726 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.233601    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:31:27.244066 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.244023    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"]
Sep 09 03:31:27.392111 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.392072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c673c-a29e-4977-aa81-c355da933c92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg\" (UID: \"130c673c-a29e-4977-aa81-c355da933c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"
Sep 09 03:31:27.493594 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.493502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c673c-a29e-4977-aa81-c355da933c92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg\" (UID: \"130c673c-a29e-4977-aa81-c355da933c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"
Sep 09 03:31:27.494133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.494115    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c673c-a29e-4977-aa81-c355da933c92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg\" (UID: \"130c673c-a29e-4977-aa81-c355da933c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"
Sep 09 03:31:27.540559 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.540528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"
Sep 09 03:31:27.820538 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:27.820504    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"]
Sep 09 03:31:27.822644 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:31:27.822620    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130c673c_a29e_4977_aa81_c355da933c92.slice/crio-9aa6b90ef09a80c23941acbb629f48c8fb393e80982d6868d9b39476c998c5f9 WatchSource:0}: Error finding container 9aa6b90ef09a80c23941acbb629f48c8fb393e80982d6868d9b39476c998c5f9: Status 404 returned error can't find the container with id 9aa6b90ef09a80c23941acbb629f48c8fb393e80982d6868d9b39476c998c5f9
Sep 09 03:31:28.567620 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:28.567586    2583 generic.go:356] "Generic (PLEG): container finished" podID="130c673c-a29e-4977-aa81-c355da933c92" containerID="1b1ed27d9c5197f07ecf618a17715929102d8c3f4ff9432bb272b80b9a4fa953" exitCode=0
Sep 09 03:31:28.568037 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:28.567668    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg" event={"ID":"130c673c-a29e-4977-aa81-c355da933c92","Type":"ContainerDied","Data":"1b1ed27d9c5197f07ecf618a17715929102d8c3f4ff9432bb272b80b9a4fa953"}
Sep 09 03:31:28.568037 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:28.567717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg" event={"ID":"130c673c-a29e-4977-aa81-c355da933c92","Type":"ContainerStarted","Data":"9aa6b90ef09a80c23941acbb629f48c8fb393e80982d6868d9b39476c998c5f9"}
Sep 09 03:31:29.711995 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:29.711958    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"
Sep 09 03:31:29.815908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:29.815879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c673c-a29e-4977-aa81-c355da933c92-maas-api-service-ca\") pod \"130c673c-a29e-4977-aa81-c355da933c92\" (UID: \"130c673c-a29e-4977-aa81-c355da933c92\") "
Sep 09 03:31:29.816300 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:29.816273    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130c673c-a29e-4977-aa81-c355da933c92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "130c673c-a29e-4977-aa81-c355da933c92" (UID: "130c673c-a29e-4977-aa81-c355da933c92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:29.917694 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:29.917664    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c673c-a29e-4977-aa81-c355da933c92-maas-api-service-ca\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.581482 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:30.581441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg" event={"ID":"130c673c-a29e-4977-aa81-c355da933c92","Type":"ContainerDied","Data":"9aa6b90ef09a80c23941acbb629f48c8fb393e80982d6868d9b39476c998c5f9"}
Sep 09 03:31:30.581660 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:30.581519    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aa6b90ef09a80c23941acbb629f48c8fb393e80982d6868d9b39476c998c5f9"
Sep 09 03:31:30.581660 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:30.581460    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"
Sep 09 03:31:43.144208 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:43.144172    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"]
Sep 09 03:31:43.147564 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:43.147530    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec5029-ts5jg"]
Sep 09 03:31:45.008204 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:45.008172    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130c673c-a29e-4977-aa81-c355da933c92" path="/var/lib/kubelet/pods/130c673c-a29e-4977-aa81-c355da933c92/volumes"
Sep 09 03:31:51.159673 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:51.159642    2583 scope.go:117] "RemoveContainer" containerID="cf804c777fccf1fa8196b8a7b71536c097c39b597efe928ac58763e8503ba7b8"
Sep 09 03:31:51.168608 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:31:51.168586    2583 scope.go:117] "RemoveContainer" containerID="2cfbddbab572ccd6a559f98d795034f48c0dd311f3cf39f6528584aec6c18b67"
Sep 09 03:32:05.643345 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:05.643297    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"]
Sep 09 03:32:05.643864 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:05.643560    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2" podUID="193e7204-e729-44a0-aace-d308f2a10d36" containerName="payload-pre-processing" containerID="cri-o://d5b6b6c6f5866bd1cf52dc25d6ddd7fc10e4724a63e641875f789836811bf4ba" gracePeriod=30
Sep 09 03:32:05.772946 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:05.772913    2583 generic.go:356] "Generic (PLEG): container finished" podID="193e7204-e729-44a0-aace-d308f2a10d36" containerID="d5b6b6c6f5866bd1cf52dc25d6ddd7fc10e4724a63e641875f789836811bf4ba" exitCode=0
Sep 09 03:32:05.773142 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:05.772995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2" event={"ID":"193e7204-e729-44a0-aace-d308f2a10d36","Type":"ContainerDied","Data":"d5b6b6c6f5866bd1cf52dc25d6ddd7fc10e4724a63e641875f789836811bf4ba"}
Sep 09 03:32:05.914349 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:05.914327    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:32:06.098317 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.098281    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e7204-e729-44a0-aace-d308f2a10d36-plugins-config-volume\") pod \"193e7204-e729-44a0-aace-d308f2a10d36\" (UID: \"193e7204-e729-44a0-aace-d308f2a10d36\") "
Sep 09 03:32:06.098510 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.098411    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w49c9\" (UniqueName: \"kubernetes.io/projected/193e7204-e729-44a0-aace-d308f2a10d36-kube-api-access-w49c9\") pod \"193e7204-e729-44a0-aace-d308f2a10d36\" (UID: \"193e7204-e729-44a0-aace-d308f2a10d36\") "
Sep 09 03:32:06.098753 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.098708    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193e7204-e729-44a0-aace-d308f2a10d36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "193e7204-e729-44a0-aace-d308f2a10d36" (UID: "193e7204-e729-44a0-aace-d308f2a10d36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:06.100796 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.100761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193e7204-e729-44a0-aace-d308f2a10d36-kube-api-access-w49c9" (OuterVolumeSpecName: "kube-api-access-w49c9") pod "193e7204-e729-44a0-aace-d308f2a10d36" (UID: "193e7204-e729-44a0-aace-d308f2a10d36"). InnerVolumeSpecName "kube-api-access-w49c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:06.199503 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.199412    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w49c9\" (UniqueName: \"kubernetes.io/projected/193e7204-e729-44a0-aace-d308f2a10d36-kube-api-access-w49c9\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:06.199503 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.199446    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e7204-e729-44a0-aace-d308f2a10d36-plugins-config-volume\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:06.779566 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.779537    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"
Sep 09 03:32:06.780036 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.779534    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2" event={"ID":"193e7204-e729-44a0-aace-d308f2a10d36","Type":"ContainerDied","Data":"7a6e2cbcc0af3ec0c5b959f6ad47413c2ec58aef598c2b41cc69515079bb1d0d"}
Sep 09 03:32:06.780036 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.779662    2583 scope.go:117] "RemoveContainer" containerID="d5b6b6c6f5866bd1cf52dc25d6ddd7fc10e4724a63e641875f789836811bf4ba"
Sep 09 03:32:06.801870 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.801846    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"]
Sep 09 03:32:06.805665 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:06.805641    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-811c96-59d97864fd-c7nx2"]
Sep 09 03:32:07.007513 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:07.007477    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="193e7204-e729-44a0-aace-d308f2a10d36" path="/var/lib/kubelet/pods/193e7204-e729-44a0-aace-d308f2a10d36/volumes"
Sep 09 03:32:10.193197 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:10.193167    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"]
Sep 09 03:32:11.190393 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:11.190354    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"]
Sep 09 03:32:12.376498 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:12.376463    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"]
Sep 09 03:32:12.376921 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:12.376758    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy" containerID="cri-o://bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91" gracePeriod=30
Sep 09 03:32:17.645960 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.645897    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:32:17.705572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705536    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-envoy\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.705572 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705577    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-podinfo\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.705819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705612    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b9912-cd9a-4563-873e-9268aa144f1e-istiod-ca-cert\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.705819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-socket\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.705819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705667    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94x98\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-kube-api-access-94x98\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.705819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705700    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-certs\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.705819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705721    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-data\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.705819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-token\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.705819 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.705786    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-credential-socket\") pod \"9b0b9912-cd9a-4563-873e-9268aa144f1e\" (UID: \"9b0b9912-cd9a-4563-873e-9268aa144f1e\") "
Sep 09 03:32:17.706147 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.706081    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:17.706231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.706193    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:17.706309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.706209    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0b9912-cd9a-4563-873e-9268aa144f1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:17.706309 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.706276    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:17.706437 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.706396    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:17.708143 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.708100    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:32:17.708272 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.708194    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:17.708272 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.708235    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-kube-api-access-94x98" (OuterVolumeSpecName: "kube-api-access-94x98") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "kube-api-access-94x98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:17.708399 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.708384    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b0b9912-cd9a-4563-873e-9268aa144f1e" (UID: "9b0b9912-cd9a-4563-873e-9268aa144f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:17.807088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807050    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94x98\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-kube-api-access-94x98\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.807088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807083    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.807088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807091    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.807088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807100    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.807359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807108    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.807359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807117    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.807359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807125    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0b9912-cd9a-4563-873e-9268aa144f1e-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.807359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807132    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b9912-cd9a-4563-873e-9268aa144f1e-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.807359 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.807140    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0b9912-cd9a-4563-873e-9268aa144f1e-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:17.842624 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.842587    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerID="bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91" exitCode=0
Sep 09 03:32:17.842764 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.842639    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"
Sep 09 03:32:17.842764 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.842677    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" event={"ID":"9b0b9912-cd9a-4563-873e-9268aa144f1e","Type":"ContainerDied","Data":"bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91"}
Sep 09 03:32:17.842880 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.842769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q" event={"ID":"9b0b9912-cd9a-4563-873e-9268aa144f1e","Type":"ContainerDied","Data":"3e4556b86dd10465420431636b9f89abd58bf89842a3931938c4a197e5325a18"}
Sep 09 03:32:17.842880 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.842797    2583 scope.go:117] "RemoveContainer" containerID="bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91"
Sep 09 03:32:17.852753 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.852731    2583 scope.go:117] "RemoveContainer" containerID="39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d"
Sep 09 03:32:17.861687 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.861668    2583 scope.go:117] "RemoveContainer" containerID="bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91"
Sep 09 03:32:17.862042 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:32:17.862021    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91\": container with ID starting with bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91 not found: ID does not exist" containerID="bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91"
Sep 09 03:32:17.862130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.862053    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91"} err="failed to get container status \"bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91\": rpc error: code = NotFound desc = could not find container \"bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91\": container with ID starting with bd6a3e3af6a78da139cd08b93c72107b82cc879e959159d6dbee5c9862eaee91 not found: ID does not exist"
Sep 09 03:32:17.862130 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.862074    2583 scope.go:117] "RemoveContainer" containerID="39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d"
Sep 09 03:32:17.862365 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:32:17.862343    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d\": container with ID starting with 39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d not found: ID does not exist" containerID="39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d"
Sep 09 03:32:17.862434 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.862379    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d"} err="failed to get container status \"39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d\": rpc error: code = NotFound desc = could not find container \"39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d\": container with ID starting with 39bed0798316fab4ac772bd2733e58f7f52bd22e585575b1766b53b9331a2f9d not found: ID does not exist"
Sep 09 03:32:17.865233 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.865207    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"]
Sep 09 03:32:17.870173 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:17.870145    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-811c96-openshift-default-6db96979f4-jcl9q"]
Sep 09 03:32:19.008979 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:19.008936    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" path="/var/lib/kubelet/pods/9b0b9912-cd9a-4563-873e-9268aa144f1e/volumes"
Sep 09 03:32:23.467237 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:23.467203    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"]
Sep 09 03:32:23.478239 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:23.478212    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-207ebd-29815410-xvcld"]
Sep 09 03:32:25.008061 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.008028    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="797d5ae8-3351-481c-aa75-075ebac647ae" path="/var/lib/kubelet/pods/797d5ae8-3351-481c-aa75-075ebac647ae/volumes"
Sep 09 03:32:25.213922 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.213878    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc" podUID="b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" containerName="main" containerID="cri-o://fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc" gracePeriod=45
Sep 09 03:32:25.473905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.473879    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:32:25.587786 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.587697    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgbfn\" (UniqueName: \"kubernetes.io/projected/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-kube-api-access-vgbfn\") pod \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") "
Sep 09 03:32:25.587786 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.587761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tmp-dir\") pod \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") "
Sep 09 03:32:25.588050 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.587801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-dshm\") pod \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") "
Sep 09 03:32:25.588050 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.587820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-home\") pod \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") "
Sep 09 03:32:25.588050 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.587880    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-model-cache\") pod \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") "
Sep 09 03:32:25.588050 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.587934    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tls-certs\") pod \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\" (UID: \"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa\") "
Sep 09 03:32:25.588265 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.588066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" (UID: "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:25.588265 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.588086    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-home" (OuterVolumeSpecName: "home") pod "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" (UID: "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:25.588265 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.588197    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" (UID: "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:25.588427 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.588316    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:25.588427 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.588352    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:25.588427 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.588368    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:25.590153 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.590126    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-kube-api-access-vgbfn" (OuterVolumeSpecName: "kube-api-access-vgbfn") pod "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" (UID: "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa"). InnerVolumeSpecName "kube-api-access-vgbfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:25.590288 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.590276    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" (UID: "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:25.590358 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.590329    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-dshm" (OuterVolumeSpecName: "dshm") pod "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" (UID: "b2539708-c61a-4c6d-a2e7-8a6bab96fbaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:25.689929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.689895    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:25.689929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.689924    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:25.689929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.689936    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgbfn\" (UniqueName: \"kubernetes.io/projected/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa-kube-api-access-vgbfn\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:25.888778 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.888746    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" containerID="fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc" exitCode=0
Sep 09 03:32:25.888972 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.888792    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"
Sep 09 03:32:25.888972 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.888792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc" event={"ID":"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa","Type":"ContainerDied","Data":"fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc"}
Sep 09 03:32:25.888972 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.888928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc" event={"ID":"b2539708-c61a-4c6d-a2e7-8a6bab96fbaa","Type":"ContainerDied","Data":"dd6e994034e1549bce4dde900c1f88dc3b220ae6ebfb7407ba286df437bfd4e9"}
Sep 09 03:32:25.889160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.888981    2583 scope.go:117] "RemoveContainer" containerID="fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc"
Sep 09 03:32:25.898721 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.898704    2583 scope.go:117] "RemoveContainer" containerID="fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc"
Sep 09 03:32:25.899005 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:32:25.898979    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc\": container with ID starting with fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc not found: ID does not exist" containerID="fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc"
Sep 09 03:32:25.899105 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.899008    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc"} err="failed to get container status \"fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc\": rpc error: code = NotFound desc = could not find container \"fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc\": container with ID starting with fb2175e6102998eb9a85f549cd09555224b39e42c08264d9deec0415abcbcddc not found: ID does not exist"
Sep 09 03:32:25.921281 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.921249    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"]
Sep 09 03:32:25.925298 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:25.925276    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1a85fa/sub-test-model-1a85fa-kserve-866dbc54c5-lprzc"]
Sep 09 03:32:26.210765 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.210671    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9" podUID="ccb6582f-431a-49a4-bfc2-30052d329ef7" containerName="main" containerID="cri-o://1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d" gracePeriod=45
Sep 09 03:32:26.459777 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.459753    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:32:26.495875 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.495783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-model-cache\") pod \"ccb6582f-431a-49a4-bfc2-30052d329ef7\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") "
Sep 09 03:32:26.496035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.495906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-tmp-dir\") pod \"ccb6582f-431a-49a4-bfc2-30052d329ef7\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") "
Sep 09 03:32:26.496035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.495941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-home\") pod \"ccb6582f-431a-49a4-bfc2-30052d329ef7\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") "
Sep 09 03:32:26.496035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.495998    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-dshm\") pod \"ccb6582f-431a-49a4-bfc2-30052d329ef7\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") "
Sep 09 03:32:26.496035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.496030    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b4hq\" (UniqueName: \"kubernetes.io/projected/ccb6582f-431a-49a4-bfc2-30052d329ef7-kube-api-access-5b4hq\") pod \"ccb6582f-431a-49a4-bfc2-30052d329ef7\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") "
Sep 09 03:32:26.496261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.496064    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb6582f-431a-49a4-bfc2-30052d329ef7-tls-certs\") pod \"ccb6582f-431a-49a4-bfc2-30052d329ef7\" (UID: \"ccb6582f-431a-49a4-bfc2-30052d329ef7\") "
Sep 09 03:32:26.496261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.496141    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-model-cache" (OuterVolumeSpecName: "model-cache") pod "ccb6582f-431a-49a4-bfc2-30052d329ef7" (UID: "ccb6582f-431a-49a4-bfc2-30052d329ef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:26.496261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.496197    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ccb6582f-431a-49a4-bfc2-30052d329ef7" (UID: "ccb6582f-431a-49a4-bfc2-30052d329ef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:26.496424 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.496303    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-home" (OuterVolumeSpecName: "home") pod "ccb6582f-431a-49a4-bfc2-30052d329ef7" (UID: "ccb6582f-431a-49a4-bfc2-30052d329ef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:26.496528 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.496500    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:26.496528 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.496529    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:26.496722 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.496542    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:26.498684 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.498657    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb6582f-431a-49a4-bfc2-30052d329ef7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ccb6582f-431a-49a4-bfc2-30052d329ef7" (UID: "ccb6582f-431a-49a4-bfc2-30052d329ef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:26.498883 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.498735    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-dshm" (OuterVolumeSpecName: "dshm") pod "ccb6582f-431a-49a4-bfc2-30052d329ef7" (UID: "ccb6582f-431a-49a4-bfc2-30052d329ef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:26.499012 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.498933    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb6582f-431a-49a4-bfc2-30052d329ef7-kube-api-access-5b4hq" (OuterVolumeSpecName: "kube-api-access-5b4hq") pod "ccb6582f-431a-49a4-bfc2-30052d329ef7" (UID: "ccb6582f-431a-49a4-bfc2-30052d329ef7"). InnerVolumeSpecName "kube-api-access-5b4hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:26.597590 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.597541    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb6582f-431a-49a4-bfc2-30052d329ef7-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:26.597590 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.597582    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb6582f-431a-49a4-bfc2-30052d329ef7-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:26.597590 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.597593    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4hq\" (UniqueName: \"kubernetes.io/projected/ccb6582f-431a-49a4-bfc2-30052d329ef7-kube-api-access-5b4hq\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:26.896317 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.896282    2583 generic.go:356] "Generic (PLEG): container finished" podID="ccb6582f-431a-49a4-bfc2-30052d329ef7" containerID="1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d" exitCode=0
Sep 09 03:32:26.896527 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.896327    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"
Sep 09 03:32:26.896527 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.896372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9" event={"ID":"ccb6582f-431a-49a4-bfc2-30052d329ef7","Type":"ContainerDied","Data":"1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d"}
Sep 09 03:32:26.896527 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.896414    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9" event={"ID":"ccb6582f-431a-49a4-bfc2-30052d329ef7","Type":"ContainerDied","Data":"b05cbee9531aa12517e2c537792aabbc6a84b2d8ac334a8d2f5da08001b31502"}
Sep 09 03:32:26.896527 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.896438    2583 scope.go:117] "RemoveContainer" containerID="1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d"
Sep 09 03:32:26.905724 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.905706    2583 scope.go:117] "RemoveContainer" containerID="1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d"
Sep 09 03:32:26.906026 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:32:26.906007    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d\": container with ID starting with 1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d not found: ID does not exist" containerID="1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d"
Sep 09 03:32:26.906102 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.906050    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d"} err="failed to get container status \"1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d\": rpc error: code = NotFound desc = could not find container \"1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d\": container with ID starting with 1bad6bddedd9d5bd6fff20d53529ecf402561afad6e93ecc16cc6a647e96d63d not found: ID does not exist"
Sep 09 03:32:26.930077 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.930022    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"]
Sep 09 03:32:26.932181 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:26.932161    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/sub-test-model-cd1da6-kserve-7d8644f59c-qblr9"]
Sep 09 03:32:27.014964 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:27.014932    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" path="/var/lib/kubelet/pods/b2539708-c61a-4c6d-a2e7-8a6bab96fbaa/volumes"
Sep 09 03:32:27.015340 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:27.015327    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb6582f-431a-49a4-bfc2-30052d329ef7" path="/var/lib/kubelet/pods/ccb6582f-431a-49a4-bfc2-30052d329ef7/volumes"
Sep 09 03:32:43.168289 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:43.168207    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"]
Sep 09 03:32:43.170205 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:43.170179    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4c8420-29815410-mwhps"]
Sep 09 03:32:45.008170 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:45.008136    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10ca3e47-2200-40da-b4f0-96e34674711f" path="/var/lib/kubelet/pods/10ca3e47-2200-40da-b4f0-96e34674711f/volumes"
Sep 09 03:32:50.450897 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:50.450863    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"]
Sep 09 03:32:50.451432 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:50.451068    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy" containerID="cri-o://ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3" gracePeriod=30
Sep 09 03:32:51.193375 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:51.193339    2583 scope.go:117] "RemoveContainer" containerID="b8b64c9a25a0436a91ea55dcd628c79f720f25804159c0b03567db2ab475c6cd"
Sep 09 03:32:51.202654 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:51.202634    2583 scope.go:117] "RemoveContainer" containerID="bd7156dd5f6c8d68f2736958f3ff5da61e6119a1339a328a2486434594b37527"
Sep 09 03:32:51.211421 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:51.211395    2583 scope.go:117] "RemoveContainer" containerID="4124bac1ad74690a843a2075b651d4881001290244d437d8de90f644358030f7"
Sep 09 03:32:51.220480 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:51.220457    2583 scope.go:117] "RemoveContainer" containerID="7761f23463cd9c0dfd8f2db556ff39e3c111de34b000fc507bb28167378f8972"
Sep 09 03:32:52.268846 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:52.268791    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:32:52.269240 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:52.268885    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:32:54.768525 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768487    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"]
Sep 09 03:32:54.768943 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768897    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="193e7204-e729-44a0-aace-d308f2a10d36" containerName="payload-pre-processing"
Sep 09 03:32:54.768943 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768911    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="193e7204-e729-44a0-aace-d308f2a10d36" containerName="payload-pre-processing"
Sep 09 03:32:54.768943 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768925    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy"
Sep 09 03:32:54.768943 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768931    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768947    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" containerName="main"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768956    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" containerName="main"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768962    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130c673c-a29e-4977-aa81-c355da933c92" containerName="revoke-keys"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768967    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="130c673c-a29e-4977-aa81-c355da933c92" containerName="revoke-keys"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768981    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb6582f-431a-49a4-bfc2-30052d329ef7" containerName="main"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.768987    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb6582f-431a-49a4-bfc2-30052d329ef7" containerName="main"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.769054    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb6582f-431a-49a4-bfc2-30052d329ef7" containerName="main"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.769061    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="130c673c-a29e-4977-aa81-c355da933c92" containerName="revoke-keys"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.769069    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.769076    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy"
Sep 09 03:32:54.769080 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.769082    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2539708-c61a-4c6d-a2e7-8a6bab96fbaa" containerName="main"
Sep 09 03:32:54.769403 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.769087    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="193e7204-e729-44a0-aace-d308f2a10d36" containerName="payload-pre-processing"
Sep 09 03:32:54.774033 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.774015    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.792437 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.792410    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"default-dockercfg-vf7q8\""
Sep 09 03:32:54.792669 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.792649    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"rate-test-model-cd1da6-kserve-self-signed-certs\""
Sep 09 03:32:54.792763 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.792747    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"kube-root-ca.crt\""
Sep 09 03:32:54.793051 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.793038    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cd1da6\"/\"openshift-service-ca.crt\""
Sep 09 03:32:54.876897 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.876823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tmp-dir\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.877092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.876910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjghp\" (UniqueName: \"kubernetes.io/projected/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-kube-api-access-pjghp\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.877092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.876960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-home\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.877092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.877024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-dshm\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.877092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.877053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-model-cache\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.877092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.877068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tls-certs\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.926266 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.926227    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"]
Sep 09 03:32:54.978696 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.978656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tmp-dir\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.978696 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.978698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjghp\" (UniqueName: \"kubernetes.io/projected/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-kube-api-access-pjghp\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.978941 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.978863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-home\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.978980 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.978943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-dshm\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.979037 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.978992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-model-cache\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.979037 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.979021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tls-certs\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.979137 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.979118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tmp-dir\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.979191 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.979160    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-home\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.979246 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.979226    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-model-cache\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.981366 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.981343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-dshm\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:54.981639 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:54.981620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tls-certs\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:55.008644 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.008615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjghp\" (UniqueName: \"kubernetes.io/projected/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-kube-api-access-pjghp\") pod \"rate-test-model-cd1da6-kserve-d857675ff-jkl6x\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") " pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:55.083537 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.083442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:32:55.368300 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.368273    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"]
Sep 09 03:32:55.370641 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:32:55.370613    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26b1879c_cf22_4a5b_ba2e_7b59e9243d50.slice/crio-6b7c8f9074c0271052ddfa375298eb65829beed1f68fed03ee0d8590a7da450e WatchSource:0}: Error finding container 6b7c8f9074c0271052ddfa375298eb65829beed1f68fed03ee0d8590a7da450e: Status 404 returned error can't find the container with id 6b7c8f9074c0271052ddfa375298eb65829beed1f68fed03ee0d8590a7da450e
Sep 09 03:32:55.700758 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.700732    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:32:55.787325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787290    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-socket\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787333    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-data\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787354    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-certs\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-credential-socket\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fed7303-16c7-472c-a928-da7dc6441174-istio-podinfo\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787450    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-envoy\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-istio-token\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787530    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fed7303-16c7-472c-a928-da7dc6441174-istiod-ca-cert\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgnvr\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-kube-api-access-dgnvr\") pod \"8fed7303-16c7-472c-a928-da7dc6441174\" (UID: \"8fed7303-16c7-472c-a928-da7dc6441174\") "
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787615    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787674    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:55.787873 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.787804    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:55.788583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.788034    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fed7303-16c7-472c-a928-da7dc6441174-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:55.788583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.788046    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:55.788583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.788066    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:55.788583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.788079    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:55.788583 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.788091    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:55.789885 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.789857    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:55.790010 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.789981    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fed7303-16c7-472c-a928-da7dc6441174-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:32:55.790010 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.789995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-kube-api-access-dgnvr" (OuterVolumeSpecName: "kube-api-access-dgnvr") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "kube-api-access-dgnvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:55.790092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.790025    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fed7303-16c7-472c-a928-da7dc6441174" (UID: "8fed7303-16c7-472c-a928-da7dc6441174"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:55.889517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.889416    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fed7303-16c7-472c-a928-da7dc6441174-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:55.889517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.889453    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgnvr\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-kube-api-access-dgnvr\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:55.889517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.889463    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fed7303-16c7-472c-a928-da7dc6441174-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:55.889517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.889473    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fed7303-16c7-472c-a928-da7dc6441174-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:55.889517 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:55.889481    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fed7303-16c7-472c-a928-da7dc6441174-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:32:56.060707 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.060671    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fed7303-16c7-472c-a928-da7dc6441174" containerID="ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3" exitCode=0
Sep 09 03:32:56.060941 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.060712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" event={"ID":"8fed7303-16c7-472c-a928-da7dc6441174","Type":"ContainerDied","Data":"ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3"}
Sep 09 03:32:56.060941 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.060733    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"
Sep 09 03:32:56.060941 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.060771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf" event={"ID":"8fed7303-16c7-472c-a928-da7dc6441174","Type":"ContainerDied","Data":"64236f315c66de4ef4a2f9767422a75591175ce565902a7eee39701818d4f8ef"}
Sep 09 03:32:56.060941 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.060795    2583 scope.go:117] "RemoveContainer" containerID="ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3"
Sep 09 03:32:56.062781 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.062757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x" event={"ID":"26b1879c-cf22-4a5b-ba2e-7b59e9243d50","Type":"ContainerStarted","Data":"ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42"}
Sep 09 03:32:56.062987 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.062798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x" event={"ID":"26b1879c-cf22-4a5b-ba2e-7b59e9243d50","Type":"ContainerStarted","Data":"6b7c8f9074c0271052ddfa375298eb65829beed1f68fed03ee0d8590a7da450e"}
Sep 09 03:32:56.070417 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.070397    2583 scope.go:117] "RemoveContainer" containerID="ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29"
Sep 09 03:32:56.079036 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.078973    2583 scope.go:117] "RemoveContainer" containerID="ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3"
Sep 09 03:32:56.079286 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:32:56.079266    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3\": container with ID starting with ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3 not found: ID does not exist" containerID="ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3"
Sep 09 03:32:56.079357 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.079295    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3"} err="failed to get container status \"ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3\": rpc error: code = NotFound desc = could not find container \"ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3\": container with ID starting with ea152ed9b38698ace639d2ecf9cd9c6363002baed8afdc1111edfb423bd446a3 not found: ID does not exist"
Sep 09 03:32:56.079357 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.079317    2583 scope.go:117] "RemoveContainer" containerID="ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29"
Sep 09 03:32:56.079572 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:32:56.079550    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29\": container with ID starting with ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29 not found: ID does not exist" containerID="ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29"
Sep 09 03:32:56.079627 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.079581    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29"} err="failed to get container status \"ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29\": rpc error: code = NotFound desc = could not find container \"ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29\": container with ID starting with ed456914bcd13f66f240df0c01a81755cabe48a7e145c1f931bf27cc496cfd29 not found: ID does not exist"
Sep 09 03:32:56.083428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.083384    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x" podStartSLOduration=2.083369926 podStartE2EDuration="2.083369926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:32:56.082887135 +0000 UTC m=+1625.596969306" watchObservedRunningTime="2026-09-09 03:32:56.083369926 +0000 UTC m=+1625.597452076"
Sep 09 03:32:56.097182 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.097155    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"]
Sep 09 03:32:56.102418 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:56.102390    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4c8420-openshift-default-cbd4bf9db-6z5wf"]
Sep 09 03:32:57.008868 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:32:57.008815    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fed7303-16c7-472c-a928-da7dc6441174" path="/var/lib/kubelet/pods/8fed7303-16c7-472c-a928-da7dc6441174/volumes"
Sep 09 03:33:05.084290 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:05.084249    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:33:05.084688 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:05.084309    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:33:05.097084 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:05.097050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:33:05.121190 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:05.121157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:33:30.893243 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:30.893209    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"]
Sep 09 03:33:31.600141 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.600108    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"]
Sep 09 03:33:31.600530 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.600516    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy"
Sep 09 03:33:31.600577 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.600532    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0b9912-cd9a-4563-873e-9268aa144f1e" containerName="istio-proxy"
Sep 09 03:33:31.600577 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.600555    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy"
Sep 09 03:33:31.600577 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.600564    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy"
Sep 09 03:33:31.600675 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.600633    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy"
Sep 09 03:33:31.600675 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.600645    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy"
Sep 09 03:33:31.604109 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.604090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.606869 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.606846    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ed81bb-openshift-default-dockercfg-g2bv2\""
Sep 09 03:33:31.622375 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.622350    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"]
Sep 09 03:33:31.634118 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100bc079-5a46-413b-8f23-0d89e2b6ea94-istiod-ca-cert\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.634230 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-envoy\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.634230 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-socket\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.634315 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tlj4\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-kube-api-access-6tlj4\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.634315 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-token\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.634315 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-podinfo\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.634425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-certs\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.634425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-credential-socket\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.634425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.634376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-data\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735063 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tlj4\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-kube-api-access-6tlj4\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-token\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-podinfo\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-certs\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-credential-socket\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735261 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-data\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735526 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100bc079-5a46-413b-8f23-0d89e2b6ea94-istiod-ca-cert\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735526 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735324    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-envoy\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735526 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-socket\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735726 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-certs\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735785 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-data\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735785 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-credential-socket\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.735915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.735869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-socket\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.736106 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.736085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100bc079-5a46-413b-8f23-0d89e2b6ea94-istiod-ca-cert\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.737872 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.737817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-envoy\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.738027 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.738012    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-podinfo\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.743930 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.743890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-token\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.744042 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.744026    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tlj4\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-kube-api-access-6tlj4\") pod \"e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") " pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:31.914672 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:31.914631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:32.195198 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:32.195136    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"]
Sep 09 03:33:32.197799 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:33:32.197772    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100bc079_5a46_413b_8f23_0d89e2b6ea94.slice/crio-b2b8beaf2381f53b25270ccc63625928fc0c1d793abcac82992a4186d3086e44 WatchSource:0}: Error finding container b2b8beaf2381f53b25270ccc63625928fc0c1d793abcac82992a4186d3086e44: Status 404 returned error can't find the container with id b2b8beaf2381f53b25270ccc63625928fc0c1d793abcac82992a4186d3086e44
Sep 09 03:33:32.200524 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:32.200490    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:33:32.200617 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:32.200560    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:33:32.200617 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:32.200588    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:33:32.262752 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:32.262721    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" event={"ID":"100bc079-5a46-413b-8f23-0d89e2b6ea94","Type":"ContainerStarted","Data":"b2b8beaf2381f53b25270ccc63625928fc0c1d793abcac82992a4186d3086e44"}
Sep 09 03:33:33.269618 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:33.269577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" event={"ID":"100bc079-5a46-413b-8f23-0d89e2b6ea94","Type":"ContainerStarted","Data":"ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d"}
Sep 09 03:33:33.296096 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:33.296041    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podStartSLOduration=2.296024455 podStartE2EDuration="2.296024455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:33:33.29365052 +0000 UTC m=+1662.807732670" watchObservedRunningTime="2026-09-09 03:33:33.296024455 +0000 UTC m=+1662.810106606"
Sep 09 03:33:33.915680 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:33.915643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:33.920934 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:33.920906    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:34.277241 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:34.277144    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:34.278187 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:34.278163    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:45.913237 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:45.913194    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x" podUID="26b1879c-cf22-4a5b-ba2e-7b59e9243d50" containerName="main" containerID="cri-o://ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42" gracePeriod=45
Sep 09 03:33:46.204798 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.204772    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:33:46.290232 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.290200    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-model-cache\") pod \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") "
Sep 09 03:33:46.290428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.290284    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-dshm\") pod \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") "
Sep 09 03:33:46.290428 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.290345    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tmp-dir\") pod \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") "
Sep 09 03:33:46.290762 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.290725    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-model-cache" (OuterVolumeSpecName: "model-cache") pod "26b1879c-cf22-4a5b-ba2e-7b59e9243d50" (UID: "26b1879c-cf22-4a5b-ba2e-7b59e9243d50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:46.290921 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.290901    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjghp\" (UniqueName: \"kubernetes.io/projected/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-kube-api-access-pjghp\") pod \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") "
Sep 09 03:33:46.291000 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.290976    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-home\") pod \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") "
Sep 09 03:33:46.291056 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.291027    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tls-certs\") pod \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\" (UID: \"26b1879c-cf22-4a5b-ba2e-7b59e9243d50\") "
Sep 09 03:33:46.291210 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.291171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26b1879c-cf22-4a5b-ba2e-7b59e9243d50" (UID: "26b1879c-cf22-4a5b-ba2e-7b59e9243d50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:46.291378 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.291356    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-home" (OuterVolumeSpecName: "home") pod "26b1879c-cf22-4a5b-ba2e-7b59e9243d50" (UID: "26b1879c-cf22-4a5b-ba2e-7b59e9243d50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:46.292216 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.292185    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-model-cache\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:33:46.292333 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.292223    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tmp-dir\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:33:46.292333 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.292246    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-home\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:33:46.302641 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.302613    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-dshm" (OuterVolumeSpecName: "dshm") pod "26b1879c-cf22-4a5b-ba2e-7b59e9243d50" (UID: "26b1879c-cf22-4a5b-ba2e-7b59e9243d50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:46.302641 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.302613    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-kube-api-access-pjghp" (OuterVolumeSpecName: "kube-api-access-pjghp") pod "26b1879c-cf22-4a5b-ba2e-7b59e9243d50" (UID: "26b1879c-cf22-4a5b-ba2e-7b59e9243d50"). InnerVolumeSpecName "kube-api-access-pjghp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:33:46.304909 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.304883    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26b1879c-cf22-4a5b-ba2e-7b59e9243d50" (UID: "26b1879c-cf22-4a5b-ba2e-7b59e9243d50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:33:46.351118 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.351078    2583 generic.go:356] "Generic (PLEG): container finished" podID="26b1879c-cf22-4a5b-ba2e-7b59e9243d50" containerID="ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42" exitCode=0
Sep 09 03:33:46.351321 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.351136    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"
Sep 09 03:33:46.351321 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.351170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x" event={"ID":"26b1879c-cf22-4a5b-ba2e-7b59e9243d50","Type":"ContainerDied","Data":"ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42"}
Sep 09 03:33:46.351321 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.351227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x" event={"ID":"26b1879c-cf22-4a5b-ba2e-7b59e9243d50","Type":"ContainerDied","Data":"6b7c8f9074c0271052ddfa375298eb65829beed1f68fed03ee0d8590a7da450e"}
Sep 09 03:33:46.351321 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.351250    2583 scope.go:117] "RemoveContainer" containerID="ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42"
Sep 09 03:33:46.360868 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.360848    2583 scope.go:117] "RemoveContainer" containerID="ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42"
Sep 09 03:33:46.361151 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:33:46.361132    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42\": container with ID starting with ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42 not found: ID does not exist" containerID="ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42"
Sep 09 03:33:46.361197 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.361159    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42"} err="failed to get container status \"ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42\": rpc error: code = NotFound desc = could not find container \"ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42\": container with ID starting with ce54f7aa26c21e1dbb528af0882769b45c1556e3ceae75dc8e0dd9ea24f55c42 not found: ID does not exist"
Sep 09 03:33:46.393449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.393415    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-dshm\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:33:46.393449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.393444    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjghp\" (UniqueName: \"kubernetes.io/projected/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-kube-api-access-pjghp\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:33:46.393449 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.393456    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26b1879c-cf22-4a5b-ba2e-7b59e9243d50-tls-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:33:46.400558 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.400530    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"]
Sep 09 03:33:46.407128 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:46.407100    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cd1da6/rate-test-model-cd1da6-kserve-d857675ff-jkl6x"]
Sep 09 03:33:47.022561 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:47.022525    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b1879c-cf22-4a5b-ba2e-7b59e9243d50" path="/var/lib/kubelet/pods/26b1879c-cf22-4a5b-ba2e-7b59e9243d50/volumes"
Sep 09 03:33:50.317995 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:50.317958    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 03:33:50.318414 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:50.318021    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 03:33:51.380104 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:51.380069    2583 generic.go:356] "Generic (PLEG): container finished" podID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerID="ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d" exitCode=0
Sep 09 03:33:51.380590 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:51.380136    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" event={"ID":"100bc079-5a46-413b-8f23-0d89e2b6ea94","Type":"ContainerDied","Data":"ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d"}
Sep 09 03:33:51.380694 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:51.380677    2583 scope.go:117] "RemoveContainer" containerID="ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d"
Sep 09 03:33:51.381905 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:51.381872    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:33:51.382016 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:51.381962    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:33:51.382067 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:51.382012    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:33:52.392022 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:52.391986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" event={"ID":"100bc079-5a46-413b-8f23-0d89e2b6ea94","Type":"ContainerStarted","Data":"b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001"}
Sep 09 03:33:52.915644 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:52.915607    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:52.915892 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:52.915869    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:33:52.915957 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:52.915924    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 03:33:53.915306 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:53.915269    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:33:53.915701 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:53.915324    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 03:33:54.915161 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:54.915126    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:33:54.915341 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:54.915183    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 03:33:55.915943 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:55.915894    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:33:55.916327 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:55.915977    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 03:33:56.915898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:56.915860    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 09 03:33:56.916095 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:56.915944    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 03:33:57.919062 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:57.919030    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:57.919535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:57.919304    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:33:57.920113 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:33:57.920096    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:34:06.662226 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.662190    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"]
Sep 09 03:34:06.662738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.662581    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy"
Sep 09 03:34:06.662738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.662592    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fed7303-16c7-472c-a928-da7dc6441174" containerName="istio-proxy"
Sep 09 03:34:06.662738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.662610    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b1879c-cf22-4a5b-ba2e-7b59e9243d50" containerName="main"
Sep 09 03:34:06.662738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.662616    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b1879c-cf22-4a5b-ba2e-7b59e9243d50" containerName="main"
Sep 09 03:34:06.662738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.662680    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b1879c-cf22-4a5b-ba2e-7b59e9243d50" containerName="main"
Sep 09 03:34:06.666097 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.666078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:06.669558 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.669534    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aacd65\""
Sep 09 03:34:06.687998 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.687962    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"]
Sep 09 03:34:06.691890 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.691866    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"]
Sep 09 03:34:06.692013 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.691926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:06.694964 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.694946    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aacd65-dockercfg-hntmb\""
Sep 09 03:34:06.697642 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.697618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38af20b8-bf92-41b6-8415-82632f5c1f42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9\" (UID: \"38af20b8-bf92-41b6-8415-82632f5c1f42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:06.697893 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.697874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sgsl\" (UniqueName: \"kubernetes.io/projected/38af20b8-bf92-41b6-8415-82632f5c1f42-kube-api-access-5sgsl\") pod \"payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9\" (UID: \"38af20b8-bf92-41b6-8415-82632f5c1f42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:06.706304 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.706229    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"]
Sep 09 03:34:06.799465 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.799309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811d65a5-42ed-40db-ad04-d539521fbc40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd\" (UID: \"811d65a5-42ed-40db-ad04-d539521fbc40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:06.799465 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.799399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62x2x\" (UniqueName: \"kubernetes.io/projected/811d65a5-42ed-40db-ad04-d539521fbc40-kube-api-access-62x2x\") pod \"payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd\" (UID: \"811d65a5-42ed-40db-ad04-d539521fbc40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:06.800200 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.799537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sgsl\" (UniqueName: \"kubernetes.io/projected/38af20b8-bf92-41b6-8415-82632f5c1f42-kube-api-access-5sgsl\") pod \"payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9\" (UID: \"38af20b8-bf92-41b6-8415-82632f5c1f42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:06.800200 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.799572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38af20b8-bf92-41b6-8415-82632f5c1f42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9\" (UID: \"38af20b8-bf92-41b6-8415-82632f5c1f42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:06.800597 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.800570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38af20b8-bf92-41b6-8415-82632f5c1f42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9\" (UID: \"38af20b8-bf92-41b6-8415-82632f5c1f42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:06.809621 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.809596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sgsl\" (UniqueName: \"kubernetes.io/projected/38af20b8-bf92-41b6-8415-82632f5c1f42-kube-api-access-5sgsl\") pod \"payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9\" (UID: \"38af20b8-bf92-41b6-8415-82632f5c1f42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:06.900421 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.900387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811d65a5-42ed-40db-ad04-d539521fbc40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd\" (UID: \"811d65a5-42ed-40db-ad04-d539521fbc40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:06.900421 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.900425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62x2x\" (UniqueName: \"kubernetes.io/projected/811d65a5-42ed-40db-ad04-d539521fbc40-kube-api-access-62x2x\") pod \"payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd\" (UID: \"811d65a5-42ed-40db-ad04-d539521fbc40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:06.901146 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.901128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811d65a5-42ed-40db-ad04-d539521fbc40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd\" (UID: \"811d65a5-42ed-40db-ad04-d539521fbc40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:06.909799 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.909763    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62x2x\" (UniqueName: \"kubernetes.io/projected/811d65a5-42ed-40db-ad04-d539521fbc40-kube-api-access-62x2x\") pod \"payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd\" (UID: \"811d65a5-42ed-40db-ad04-d539521fbc40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:06.976507 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:06.976424    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:07.002739 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.002676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:07.260302 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.260271    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"]
Sep 09 03:34:07.262278 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:34:07.262249    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811d65a5_42ed_40db_ad04_d539521fbc40.slice/crio-a244feb2e85cc5ba926feb8b3756b06111fd2355642cfc6a2860b549a726c7d4 WatchSource:0}: Error finding container a244feb2e85cc5ba926feb8b3756b06111fd2355642cfc6a2860b549a726c7d4: Status 404 returned error can't find the container with id a244feb2e85cc5ba926feb8b3756b06111fd2355642cfc6a2860b549a726c7d4
Sep 09 03:34:07.289950 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.289924    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"]
Sep 09 03:34:07.295134 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:34:07.295103    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38af20b8_bf92_41b6_8415_82632f5c1f42.slice/crio-32a3b7ba3054ac843346f893b9317fd44015749ccde962d9a0028d9d8908829d WatchSource:0}: Error finding container 32a3b7ba3054ac843346f893b9317fd44015749ccde962d9a0028d9d8908829d: Status 404 returned error can't find the container with id 32a3b7ba3054ac843346f893b9317fd44015749ccde962d9a0028d9d8908829d
Sep 09 03:34:07.474558 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.474517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd" event={"ID":"811d65a5-42ed-40db-ad04-d539521fbc40","Type":"ContainerStarted","Data":"4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e"}
Sep 09 03:34:07.474765 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.474590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd" event={"ID":"811d65a5-42ed-40db-ad04-d539521fbc40","Type":"ContainerStarted","Data":"a244feb2e85cc5ba926feb8b3756b06111fd2355642cfc6a2860b549a726c7d4"}
Sep 09 03:34:07.474765 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.474621    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:07.476734 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.476708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9" event={"ID":"38af20b8-bf92-41b6-8415-82632f5c1f42","Type":"ContainerStarted","Data":"2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2"}
Sep 09 03:34:07.476894 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.476755    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9" event={"ID":"38af20b8-bf92-41b6-8415-82632f5c1f42","Type":"ContainerStarted","Data":"32a3b7ba3054ac843346f893b9317fd44015749ccde962d9a0028d9d8908829d"}
Sep 09 03:34:07.476894 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.476803    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:07.507935 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.507867    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd" podStartSLOduration=1.507845275 podStartE2EDuration="1.507845275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:34:07.505674708 +0000 UTC m=+1697.019756882" watchObservedRunningTime="2026-09-09 03:34:07.507845275 +0000 UTC m=+1697.021927418"
Sep 09 03:34:07.540349 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:07.540242    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9" podStartSLOduration=1.540178636 podStartE2EDuration="1.540178636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:34:07.539946364 +0000 UTC m=+1697.054028536" watchObservedRunningTime="2026-09-09 03:34:07.540178636 +0000 UTC m=+1697.054260822"
Sep 09 03:34:18.484497 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:18.484412    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:34:18.484497 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:18.484487    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:34:35.242283 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:35.242250    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9_38af20b8-bf92-41b6-8415-82632f5c1f42/payload-processing/0.log"
Sep 09 03:34:51.290535 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:34:51.290499    2583 scope.go:117] "RemoveContainer" containerID="3c18fb9f57faef41ffddf75153b2faff03cfdd1c0649a9c3e790b367ca557e7b"
Sep 09 03:35:06.117543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.117516    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9_38af20b8-bf92-41b6-8415-82632f5c1f42/payload-processing/0.log"
Sep 09 03:35:06.754850 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.754797    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"]
Sep 09 03:35:06.759750 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.759729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.762298 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.762279    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-7e9fa8-openshift-default-dockercfg-wdccg\""
Sep 09 03:35:06.769941 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.769916    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"]
Sep 09 03:35:06.872641 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.872606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-data\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.872865 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.872676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-token\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.872865 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.872725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-credential-socket\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.872865 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.872807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-envoy\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.873053 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.872881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-socket\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.873053 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.872928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-certs\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.873053 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.872957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05587df9-4704-4dc9-a752-16166c5a4f8d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.873053 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.872981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6wb\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-kube-api-access-ss6wb\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.873231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.873118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-podinfo\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.974464 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-credential-socket\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.974725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-envoy\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.974725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-socket\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.974725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-certs\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.974725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05587df9-4704-4dc9-a752-16166c5a4f8d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.974725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6wb\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-kube-api-access-ss6wb\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.974725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-podinfo\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.975081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-data\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.975081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-token\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.975081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-socket\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.975081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.974910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-credential-socket\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.975378 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.975358    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-data\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.975432 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.975387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-certs\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.975589 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.975567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05587df9-4704-4dc9-a752-16166c5a4f8d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.977122 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.977103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-envoy\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.977390 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.977365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-podinfo\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.986505 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.986477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-token\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:06.986599 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:06.986510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6wb\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-kube-api-access-ss6wb\") pod \"e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") " pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:07.071191 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:07.071087    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:07.349112 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:07.349088    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"]
Sep 09 03:35:07.350098 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:35:07.350076    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05587df9_4704_4dc9_a752_16166c5a4f8d.slice/crio-537c6cb59459a6b04cc84c70b170bb31060c23a113a30689be47072457dddb98 WatchSource:0}: Error finding container 537c6cb59459a6b04cc84c70b170bb31060c23a113a30689be47072457dddb98: Status 404 returned error can't find the container with id 537c6cb59459a6b04cc84c70b170bb31060c23a113a30689be47072457dddb98
Sep 09 03:35:07.352277 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:07.352244    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:35:07.352356 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:07.352342    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:35:07.352395 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:07.352381    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:35:07.805679 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:07.805639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" event={"ID":"05587df9-4704-4dc9-a752-16166c5a4f8d","Type":"ContainerStarted","Data":"992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559"}
Sep 09 03:35:07.805895 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:07.805705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" event={"ID":"05587df9-4704-4dc9-a752-16166c5a4f8d","Type":"ContainerStarted","Data":"537c6cb59459a6b04cc84c70b170bb31060c23a113a30689be47072457dddb98"}
Sep 09 03:35:07.830903 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:07.830849    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" podStartSLOduration=1.830812117 podStartE2EDuration="1.830812117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:07.829819486 +0000 UTC m=+1757.343901637" watchObservedRunningTime="2026-09-09 03:35:07.830812117 +0000 UTC m=+1757.344894267"
Sep 09 03:35:08.071695 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:08.071597    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:08.077001 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:08.076972    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:08.811444 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:08.811408    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:08.812512 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:08.812494    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:30.945798 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:30.945761    2583 generic.go:356] "Generic (PLEG): container finished" podID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerID="992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559" exitCode=0
Sep 09 03:35:30.946341 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:30.945862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" event={"ID":"05587df9-4704-4dc9-a752-16166c5a4f8d","Type":"ContainerDied","Data":"992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559"}
Sep 09 03:35:30.946414 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:30.946398    2583 scope.go:117] "RemoveContainer" containerID="992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559"
Sep 09 03:35:30.947554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:30.947525    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:35:30.947659 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:30.947611    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:35:30.947727 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:30.947658    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 03:35:31.952816 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:31.952780    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" event={"ID":"05587df9-4704-4dc9-a752-16166c5a4f8d","Type":"ContainerStarted","Data":"e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef"}
Sep 09 03:35:32.072267 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:32.072224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:32.072476 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:32.072361    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 09 03:35:32.072476 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:32.072420    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 09 03:35:33.071613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:33.071571    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 09 03:35:33.072162 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:33.071657    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 09 03:35:34.071803 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:34.071761    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 09 03:35:34.072470 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:34.072435    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 09 03:35:35.073021 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:35.072526    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 09 03:35:35.073021 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:35.072602    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 09 03:35:36.072267 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:36.072230    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 09 03:35:36.072586 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:36.072546    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 09 03:35:37.180088 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:37.180058    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:37.180643 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:37.180415    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:37.181414 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:37.181398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:48.588401 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:48.588362    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"]
Sep 09 03:35:48.588792 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:48.588640    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy" containerID="cri-o://e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef" gracePeriod=30
Sep 09 03:35:51.053313 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:51.053282    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:35:51.056182 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:51.056154    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:35:51.059370 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:51.059352    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:35:51.062422 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:51.062391    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:35:52.182637 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:52.182597    2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:35:52.183070 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:52.182665    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:35:53.834633 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.834602    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:53.875560 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.875513    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05587df9-4704-4dc9-a752-16166c5a4f8d-istiod-ca-cert\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.875718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.875615    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-podinfo\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.875718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.875645    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-certs\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.875718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.875693    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-credential-socket\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.875937 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.875737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-token\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.875937 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.875920    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:35:53.876044 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.875947    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05587df9-4704-4dc9-a752-16166c5a4f8d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:35:53.876102 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.876047    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:35:53.876228 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.876207    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05587df9-4704-4dc9-a752-16166c5a4f8d-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:53.876228 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.876227    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:53.876420 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.876236    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:53.878022 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.877998    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-token" (OuterVolumeSpecName: "istio-token") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:35:53.878022 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.878009    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:35:53.976725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.976690    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss6wb\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-kube-api-access-ss6wb\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.976725 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.976737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-socket\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.977005 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.976761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-data\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.977005 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.976911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-envoy\") pod \"05587df9-4704-4dc9-a752-16166c5a4f8d\" (UID: \"05587df9-4704-4dc9-a752-16166c5a4f8d\") "
Sep 09 03:35:53.977115 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.977049    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:35:53.977115 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.977063    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-data" (OuterVolumeSpecName: "istio-data") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:35:53.977225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.977199    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:53.977225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.977213    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:53.977225 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.977223    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:53.977343 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.977231    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:53.979118 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.979094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:35:53.979221 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:53.979131    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-kube-api-access-ss6wb" (OuterVolumeSpecName: "kube-api-access-ss6wb") pod "05587df9-4704-4dc9-a752-16166c5a4f8d" (UID: "05587df9-4704-4dc9-a752-16166c5a4f8d"). InnerVolumeSpecName "kube-api-access-ss6wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:35:54.077959 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.077900    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05587df9-4704-4dc9-a752-16166c5a4f8d-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:54.077959 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.077953    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss6wb\" (UniqueName: \"kubernetes.io/projected/05587df9-4704-4dc9-a752-16166c5a4f8d-kube-api-access-ss6wb\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:35:54.085718 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.085685    2583 generic.go:356] "Generic (PLEG): container finished" podID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerID="e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef" exitCode=0
Sep 09 03:35:54.085929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.085734    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"
Sep 09 03:35:54.085929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.085772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" event={"ID":"05587df9-4704-4dc9-a752-16166c5a4f8d","Type":"ContainerDied","Data":"e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef"}
Sep 09 03:35:54.085929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.085849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2" event={"ID":"05587df9-4704-4dc9-a752-16166c5a4f8d","Type":"ContainerDied","Data":"537c6cb59459a6b04cc84c70b170bb31060c23a113a30689be47072457dddb98"}
Sep 09 03:35:54.085929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.085879    2583 scope.go:117] "RemoveContainer" containerID="e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef"
Sep 09 03:35:54.095716 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.095697    2583 scope.go:117] "RemoveContainer" containerID="992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559"
Sep 09 03:35:54.104425 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.104404    2583 scope.go:117] "RemoveContainer" containerID="e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef"
Sep 09 03:35:54.104846 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:35:54.104759    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef\": container with ID starting with e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef not found: ID does not exist" containerID="e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef"
Sep 09 03:35:54.104846 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.104801    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef"} err="failed to get container status \"e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef\": rpc error: code = NotFound desc = could not find container \"e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef\": container with ID starting with e1c32645a6c3454c10c929793f3f81a812189a4118dadbe8e922a39a73675eef not found: ID does not exist"
Sep 09 03:35:54.105045 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.104851    2583 scope.go:117] "RemoveContainer" containerID="992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559"
Sep 09 03:35:54.105591 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:35:54.105570    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559\": container with ID starting with 992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559 not found: ID does not exist" containerID="992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559"
Sep 09 03:35:54.105695 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.105601    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559"} err="failed to get container status \"992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559\": rpc error: code = NotFound desc = could not find container \"992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559\": container with ID starting with 992e0a9ea39a1cc6c4728c238b429bf2e235a4e3c8389f68ed5a8fd926474559 not found: ID does not exist"
Sep 09 03:35:54.107223 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.107204    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"]
Sep 09 03:35:54.110681 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:54.110660    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7e9fa8-openshift-default-5847b48bbd-tjwf2"]
Sep 09 03:35:55.008128 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:35:55.008093    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" path="/var/lib/kubelet/pods/05587df9-4704-4dc9-a752-16166c5a4f8d/volumes"
Sep 09 03:36:51.319949 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:36:51.319914    2583 scope.go:117] "RemoveContainer" containerID="d703f7bd1c16ef2fcf0f88992d1d904cb389784151a7dc876991c87829862aaf"
Sep 09 03:36:51.328871 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:36:51.328848    2583 scope.go:117] "RemoveContainer" containerID="440fdf02bffa1039d8370cccbd93aa5465f87440e78eca01b6b99350c91392ae"
Sep 09 03:36:57.030322 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:36:57.030286    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"]
Sep 09 03:36:57.030782 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:36:57.030548    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy" containerID="cri-o://b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001" gracePeriod=30
Sep 09 03:36:57.922182 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:36:57.922146    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:36:57.922351 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:36:57.922204    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:37:02.280546 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.280516    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:37:02.435885 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.435822    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-podinfo\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436089 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.435944    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-socket\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436089 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.435969    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-data\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436089 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436015    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-token\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436089 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436041    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-credential-socket\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436089 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436080    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tlj4\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-kube-api-access-6tlj4\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100bc079-5a46-413b-8f23-0d89e2b6ea94-istiod-ca-cert\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-certs\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436162    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-envoy\") pod \"100bc079-5a46-413b-8f23-0d89e2b6ea94\" (UID: \"100bc079-5a46-413b-8f23-0d89e2b6ea94\") "
Sep 09 03:37:02.436361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:02.436361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-data" (OuterVolumeSpecName: "istio-data") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:02.436361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436353    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:02.436685 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436517    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:02.436685 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436594    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:02.436685 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436611    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:02.436685 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436623    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:02.436685 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436632    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:02.437164 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.436718    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100bc079-5a46-413b-8f23-0d89e2b6ea94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:02.438308 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.438280    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:37:02.438440 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.438403    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:02.438506 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.438478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-token" (OuterVolumeSpecName: "istio-token") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:02.438820 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.438801    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-kube-api-access-6tlj4" (OuterVolumeSpecName: "kube-api-access-6tlj4") pod "100bc079-5a46-413b-8f23-0d89e2b6ea94" (UID: "100bc079-5a46-413b-8f23-0d89e2b6ea94"). InnerVolumeSpecName "kube-api-access-6tlj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:02.472865 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.472808    2583 generic.go:356] "Generic (PLEG): container finished" podID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerID="b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001" exitCode=0
Sep 09 03:37:02.473032 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.472863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" event={"ID":"100bc079-5a46-413b-8f23-0d89e2b6ea94","Type":"ContainerDied","Data":"b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001"}
Sep 09 03:37:02.473032 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.472890    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"
Sep 09 03:37:02.473032 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.472930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl" event={"ID":"100bc079-5a46-413b-8f23-0d89e2b6ea94","Type":"ContainerDied","Data":"b2b8beaf2381f53b25270ccc63625928fc0c1d793abcac82992a4186d3086e44"}
Sep 09 03:37:02.473032 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.472960    2583 scope.go:117] "RemoveContainer" containerID="b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001"
Sep 09 03:37:02.482645 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.482627    2583 scope.go:117] "RemoveContainer" containerID="ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d"
Sep 09 03:37:02.490560 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.490542    2583 scope.go:117] "RemoveContainer" containerID="b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001"
Sep 09 03:37:02.490861 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:37:02.490824    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001\": container with ID starting with b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001 not found: ID does not exist" containerID="b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001"
Sep 09 03:37:02.490914 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.490871    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001"} err="failed to get container status \"b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001\": rpc error: code = NotFound desc = could not find container \"b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001\": container with ID starting with b301d3d4e1a82896643760aac793e9a8ec6c66e9cf9883843b1d775b11a27001 not found: ID does not exist"
Sep 09 03:37:02.490914 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.490889    2583 scope.go:117] "RemoveContainer" containerID="ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d"
Sep 09 03:37:02.491145 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:37:02.491124    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d\": container with ID starting with ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d not found: ID does not exist" containerID="ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d"
Sep 09 03:37:02.491207 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.491155    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d"} err="failed to get container status \"ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d\": rpc error: code = NotFound desc = could not find container \"ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d\": container with ID starting with ff7a8ca8abb98fc8cc1c4751274c3e0c00158f4b48510434a8783971b90df64d not found: ID does not exist"
Sep 09 03:37:02.494119 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.494095    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"]
Sep 09 03:37:02.498433 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.498413    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed81bb-openshift-default-64674f665d-q97fl"]
Sep 09 03:37:02.537917 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.537882    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:02.537917 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.537915    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:02.538133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.537929    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tlj4\" (UniqueName: \"kubernetes.io/projected/100bc079-5a46-413b-8f23-0d89e2b6ea94-kube-api-access-6tlj4\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:02.538133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.537942    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100bc079-5a46-413b-8f23-0d89e2b6ea94-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:02.538133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:02.537953    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100bc079-5a46-413b-8f23-0d89e2b6ea94-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:03.007787 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:03.007751    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" path="/var/lib/kubelet/pods/100bc079-5a46-413b-8f23-0d89e2b6ea94/volumes"
Sep 09 03:37:08.179007 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.178968    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"]
Sep 09 03:37:08.179479 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.179189    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9" podUID="38af20b8-bf92-41b6-8415-82632f5c1f42" containerName="payload-processing" containerID="cri-o://2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2" gracePeriod=30
Sep 09 03:37:08.201222 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.201129    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"]
Sep 09 03:37:08.443974 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.443901    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:37:08.505077 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.505037    2583 generic.go:356] "Generic (PLEG): container finished" podID="38af20b8-bf92-41b6-8415-82632f5c1f42" containerID="2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2" exitCode=0
Sep 09 03:37:08.505246 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.505088    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9" event={"ID":"38af20b8-bf92-41b6-8415-82632f5c1f42","Type":"ContainerDied","Data":"2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2"}
Sep 09 03:37:08.505246 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.505120    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"
Sep 09 03:37:08.505246 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.505136    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9" event={"ID":"38af20b8-bf92-41b6-8415-82632f5c1f42","Type":"ContainerDied","Data":"32a3b7ba3054ac843346f893b9317fd44015749ccde962d9a0028d9d8908829d"}
Sep 09 03:37:08.505246 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.505157    2583 scope.go:117] "RemoveContainer" containerID="2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2"
Sep 09 03:37:08.505486 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.505306    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd" podUID="811d65a5-42ed-40db-ad04-d539521fbc40" containerName="payload-pre-processing" containerID="cri-o://4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e" gracePeriod=30
Sep 09 03:37:08.514782 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.514759    2583 scope.go:117] "RemoveContainer" containerID="2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2"
Sep 09 03:37:08.515109 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:37:08.515088    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2\": container with ID starting with 2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2 not found: ID does not exist" containerID="2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2"
Sep 09 03:37:08.515157 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.515119    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2"} err="failed to get container status \"2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2\": rpc error: code = NotFound desc = could not find container \"2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2\": container with ID starting with 2b9e1bae4bc5ec218c837ddeb7621e47280d63e4704a1110d6093ce320af51f2 not found: ID does not exist"
Sep 09 03:37:08.596708 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.596675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38af20b8-bf92-41b6-8415-82632f5c1f42-plugins-config-volume\") pod \"38af20b8-bf92-41b6-8415-82632f5c1f42\" (UID: \"38af20b8-bf92-41b6-8415-82632f5c1f42\") "
Sep 09 03:37:08.596886 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.596754    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sgsl\" (UniqueName: \"kubernetes.io/projected/38af20b8-bf92-41b6-8415-82632f5c1f42-kube-api-access-5sgsl\") pod \"38af20b8-bf92-41b6-8415-82632f5c1f42\" (UID: \"38af20b8-bf92-41b6-8415-82632f5c1f42\") "
Sep 09 03:37:08.597107 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.597075    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38af20b8-bf92-41b6-8415-82632f5c1f42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38af20b8-bf92-41b6-8415-82632f5c1f42" (UID: "38af20b8-bf92-41b6-8415-82632f5c1f42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:08.599113 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.599086    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38af20b8-bf92-41b6-8415-82632f5c1f42-kube-api-access-5sgsl" (OuterVolumeSpecName: "kube-api-access-5sgsl") pod "38af20b8-bf92-41b6-8415-82632f5c1f42" (UID: "38af20b8-bf92-41b6-8415-82632f5c1f42"). InnerVolumeSpecName "kube-api-access-5sgsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:08.698423 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.698354    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sgsl\" (UniqueName: \"kubernetes.io/projected/38af20b8-bf92-41b6-8415-82632f5c1f42-kube-api-access-5sgsl\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:08.698423 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.698389    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38af20b8-bf92-41b6-8415-82632f5c1f42-plugins-config-volume\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:08.752365 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.752340    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:37:08.825990 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.825958    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"]
Sep 09 03:37:08.829711 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.829681    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aacd65-6d8ffcc78c-r67c9"]
Sep 09 03:37:08.900280 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.900246    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811d65a5-42ed-40db-ad04-d539521fbc40-plugins-config-volume\") pod \"811d65a5-42ed-40db-ad04-d539521fbc40\" (UID: \"811d65a5-42ed-40db-ad04-d539521fbc40\") "
Sep 09 03:37:08.900280 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.900284    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62x2x\" (UniqueName: \"kubernetes.io/projected/811d65a5-42ed-40db-ad04-d539521fbc40-kube-api-access-62x2x\") pod \"811d65a5-42ed-40db-ad04-d539521fbc40\" (UID: \"811d65a5-42ed-40db-ad04-d539521fbc40\") "
Sep 09 03:37:08.900698 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.900665    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811d65a5-42ed-40db-ad04-d539521fbc40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "811d65a5-42ed-40db-ad04-d539521fbc40" (UID: "811d65a5-42ed-40db-ad04-d539521fbc40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:08.902691 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:08.902668    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811d65a5-42ed-40db-ad04-d539521fbc40-kube-api-access-62x2x" (OuterVolumeSpecName: "kube-api-access-62x2x") pod "811d65a5-42ed-40db-ad04-d539521fbc40" (UID: "811d65a5-42ed-40db-ad04-d539521fbc40"). InnerVolumeSpecName "kube-api-access-62x2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:09.001953 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.001864    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811d65a5-42ed-40db-ad04-d539521fbc40-plugins-config-volume\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:09.002104 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.001994    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62x2x\" (UniqueName: \"kubernetes.io/projected/811d65a5-42ed-40db-ad04-d539521fbc40-kube-api-access-62x2x\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:09.009028 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.008269    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38af20b8-bf92-41b6-8415-82632f5c1f42" path="/var/lib/kubelet/pods/38af20b8-bf92-41b6-8415-82632f5c1f42/volumes"
Sep 09 03:37:09.512070 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.512032    2583 generic.go:356] "Generic (PLEG): container finished" podID="811d65a5-42ed-40db-ad04-d539521fbc40" containerID="4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e" exitCode=0
Sep 09 03:37:09.512553 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.512089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd" event={"ID":"811d65a5-42ed-40db-ad04-d539521fbc40","Type":"ContainerDied","Data":"4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e"}
Sep 09 03:37:09.512553 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.512095    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"
Sep 09 03:37:09.512553 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.512131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd" event={"ID":"811d65a5-42ed-40db-ad04-d539521fbc40","Type":"ContainerDied","Data":"a244feb2e85cc5ba926feb8b3756b06111fd2355642cfc6a2860b549a726c7d4"}
Sep 09 03:37:09.512553 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.512152    2583 scope.go:117] "RemoveContainer" containerID="4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e"
Sep 09 03:37:09.520742 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.520725    2583 scope.go:117] "RemoveContainer" containerID="4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e"
Sep 09 03:37:09.521048 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:37:09.521028    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e\": container with ID starting with 4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e not found: ID does not exist" containerID="4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e"
Sep 09 03:37:09.521123 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.521061    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e"} err="failed to get container status \"4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e\": rpc error: code = NotFound desc = could not find container \"4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e\": container with ID starting with 4a5ae12e65c5b1f607bed449583607c91ae57151ace7f2c1d2dd2d1b1a160a0e not found: ID does not exist"
Sep 09 03:37:09.549524 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.549492    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"]
Sep 09 03:37:09.569927 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:09.569886    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aacd65-7fb5b7d6cd-nz6bd"]
Sep 09 03:37:11.007023 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:11.006991    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="811d65a5-42ed-40db-ad04-d539521fbc40" path="/var/lib/kubelet/pods/811d65a5-42ed-40db-ad04-d539521fbc40/volumes"
Sep 09 03:37:50.677197 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:50.677162    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"]
Sep 09 03:37:50.677636 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:50.677433    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy" containerID="cri-o://ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61" gracePeriod=30
Sep 09 03:37:50.712433 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:50.712394    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:37:50.712600 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:50.712475    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:37:51.355966 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:51.355933    2583 scope.go:117] "RemoveContainer" containerID="1b1ed27d9c5197f07ecf618a17715929102d8c3f4ff9432bb272b80b9a4fa953"
Sep 09 03:37:53.361494 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.361456    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-vs7gh"]
Sep 09 03:37:53.361946 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.361642    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66bcff4f74-vs7gh" podUID="b3da7eac-523d-4d83-8da7-8cbf9b5e583a" containerName="manager" containerID="cri-o://d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a" gracePeriod=30
Sep 09 03:37:53.614366 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.614302    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:37:53.627405 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.627380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmzs7\" (UniqueName: \"kubernetes.io/projected/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-kube-api-access-bmzs7\") pod \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") "
Sep 09 03:37:53.627502 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.627415    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-webhook-cert\") pod \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") "
Sep 09 03:37:53.627502 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.627488    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-metrics-certs\") pod \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\" (UID: \"b3da7eac-523d-4d83-8da7-8cbf9b5e583a\") "
Sep 09 03:37:53.629676 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.629642    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-kube-api-access-bmzs7" (OuterVolumeSpecName: "kube-api-access-bmzs7") pod "b3da7eac-523d-4d83-8da7-8cbf9b5e583a" (UID: "b3da7eac-523d-4d83-8da7-8cbf9b5e583a"). InnerVolumeSpecName "kube-api-access-bmzs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:53.629789 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.629708    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3da7eac-523d-4d83-8da7-8cbf9b5e583a" (UID: "b3da7eac-523d-4d83-8da7-8cbf9b5e583a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:53.629789 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.629771    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b3da7eac-523d-4d83-8da7-8cbf9b5e583a" (UID: "b3da7eac-523d-4d83-8da7-8cbf9b5e583a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:53.728702 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.728656    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-metrics-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:53.728702 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.728696    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzs7\" (UniqueName: \"kubernetes.io/projected/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-kube-api-access-bmzs7\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:53.728958 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.728713    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3da7eac-523d-4d83-8da7-8cbf9b5e583a-webhook-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:53.739043 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.739008    2583 generic.go:356] "Generic (PLEG): container finished" podID="b3da7eac-523d-4d83-8da7-8cbf9b5e583a" containerID="d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a" exitCode=0
Sep 09 03:37:53.739220 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.739048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-vs7gh" event={"ID":"b3da7eac-523d-4d83-8da7-8cbf9b5e583a","Type":"ContainerDied","Data":"d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a"}
Sep 09 03:37:53.739220 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.739068    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-vs7gh"
Sep 09 03:37:53.739220 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.739120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-vs7gh" event={"ID":"b3da7eac-523d-4d83-8da7-8cbf9b5e583a","Type":"ContainerDied","Data":"74dabfcb998f4fed4bde99bca9257e354ba353121282ed59517a6a7be4009aae"}
Sep 09 03:37:53.739220 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.739138    2583 scope.go:117] "RemoveContainer" containerID="d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a"
Sep 09 03:37:53.749755 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.749735    2583 scope.go:117] "RemoveContainer" containerID="d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a"
Sep 09 03:37:53.750043 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:37:53.750024    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a\": container with ID starting with d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a not found: ID does not exist" containerID="d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a"
Sep 09 03:37:53.750096 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.750054    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a"} err="failed to get container status \"d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a\": rpc error: code = NotFound desc = could not find container \"d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a\": container with ID starting with d524440fece43595ae9f26c2e56a1a5dc72b38563649a8422db401634fe2479a not found: ID does not exist"
Sep 09 03:37:53.759792 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.759768    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-vs7gh"]
Sep 09 03:37:53.764096 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:53.764075    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-vs7gh"]
Sep 09 03:37:55.013305 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:55.013268    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3da7eac-523d-4d83-8da7-8cbf9b5e583a" path="/var/lib/kubelet/pods/b3da7eac-523d-4d83-8da7-8cbf9b5e583a/volumes"
Sep 09 03:37:55.939379 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:55.939356    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:37:56.051273 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051161    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-socket\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051273 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051218    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-envoy\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051273 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051246    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-certs\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051273 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051271    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpkxs\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-kube-api-access-vpkxs\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051315    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-istio-token\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051373    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-data\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051409    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-credential-socket\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051482    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051494    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c67ea8-e39e-499c-965a-369909fd1478-istiod-ca-cert\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051557    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c67ea8-e39e-499c-965a-369909fd1478-istio-podinfo\") pod \"29c67ea8-e39e-499c-965a-369909fd1478\" (UID: \"29c67ea8-e39e-499c-965a-369909fd1478\") "
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051636    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051824    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:56.051977 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.051937    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c67ea8-e39e-499c-965a-369909fd1478-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:56.052455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.052041    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.052455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.052063    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-workload-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.052455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.052077    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-credential-socket\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.052455 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.052091    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c67ea8-e39e-499c-965a-369909fd1478-istiod-ca-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.052620 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.052476    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-data" (OuterVolumeSpecName: "istio-data") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:56.053844 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.053808    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-istio-token" (OuterVolumeSpecName: "istio-token") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:56.053950 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.053883    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29c67ea8-e39e-499c-965a-369909fd1478-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:37:56.054245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.054226    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:37:56.054245 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.054233    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-kube-api-access-vpkxs" (OuterVolumeSpecName: "kube-api-access-vpkxs") pod "29c67ea8-e39e-499c-965a-369909fd1478" (UID: "29c67ea8-e39e-499c-965a-369909fd1478"). InnerVolumeSpecName "kube-api-access-vpkxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:56.152938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.152896    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-data\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.152938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.152931    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c67ea8-e39e-499c-965a-369909fd1478-istio-podinfo\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.152938 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.152943    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c67ea8-e39e-499c-965a-369909fd1478-istio-envoy\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.153182 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.152958    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkxs\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-kube-api-access-vpkxs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.153182 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.152971    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c67ea8-e39e-499c-965a-369909fd1478-istio-token\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:37:56.758686 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.758646    2583 generic.go:356] "Generic (PLEG): container finished" podID="29c67ea8-e39e-499c-965a-369909fd1478" containerID="ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61" exitCode=0
Sep 09 03:37:56.758898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.758696    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"
Sep 09 03:37:56.758898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.758733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" event={"ID":"29c67ea8-e39e-499c-965a-369909fd1478","Type":"ContainerDied","Data":"ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61"}
Sep 09 03:37:56.758898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.758792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92" event={"ID":"29c67ea8-e39e-499c-965a-369909fd1478","Type":"ContainerDied","Data":"788cebda89db8c3785ce1bb2c4afd22979bcb5b54667227e108bc242ebfc8622"}
Sep 09 03:37:56.758898 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.758819    2583 scope.go:117] "RemoveContainer" containerID="ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61"
Sep 09 03:37:56.768244 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.768224    2583 scope.go:117] "RemoveContainer" containerID="f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898"
Sep 09 03:37:56.776762 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.776747    2583 scope.go:117] "RemoveContainer" containerID="ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61"
Sep 09 03:37:56.777041 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:37:56.777025    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61\": container with ID starting with ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61 not found: ID does not exist" containerID="ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61"
Sep 09 03:37:56.777086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.777051    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61"} err="failed to get container status \"ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61\": rpc error: code = NotFound desc = could not find container \"ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61\": container with ID starting with ae315a2627ab0538378da368f10f3da69fef11bdf59cd1b567c08df110f07a61 not found: ID does not exist"
Sep 09 03:37:56.777086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.777066    2583 scope.go:117] "RemoveContainer" containerID="f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898"
Sep 09 03:37:56.777305 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:37:56.777291    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898\": container with ID starting with f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898 not found: ID does not exist" containerID="f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898"
Sep 09 03:37:56.777342 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.777310    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898"} err="failed to get container status \"f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898\": rpc error: code = NotFound desc = could not find container \"f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898\": container with ID starting with f66263878f5650c4219a6a0abb60abee20633a50a01a01f8f91d3abe21618898 not found: ID does not exist"
Sep 09 03:37:56.799357 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.799324    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"]
Sep 09 03:37:56.803086 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:56.803062    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-cd1da6-openshift-default-8659564584-4nd92"]
Sep 09 03:37:57.007160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:37:57.007127    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c67ea8-e39e-499c-965a-369909fd1478" path="/var/lib/kubelet/pods/29c67ea8-e39e-499c-965a-369909fd1478/volumes"
Sep 09 03:38:54.591676 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.591635    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66bcff4f74-d9jks"]
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592049    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38af20b8-bf92-41b6-8415-82632f5c1f42" containerName="payload-processing"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592063    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="38af20b8-bf92-41b6-8415-82632f5c1f42" containerName="payload-processing"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592071    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592077    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592085    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3da7eac-523d-4d83-8da7-8cbf9b5e583a" containerName="manager"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592091    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3da7eac-523d-4d83-8da7-8cbf9b5e583a" containerName="manager"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592097    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592103    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592116    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592121    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592127    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="811d65a5-42ed-40db-ad04-d539521fbc40" containerName="payload-pre-processing"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592132    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="811d65a5-42ed-40db-ad04-d539521fbc40" containerName="payload-pre-processing"
Sep 09 03:38:54.592134 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592140    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592145    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592153    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592158    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592211    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592218    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="38af20b8-bf92-41b6-8415-82632f5c1f42" containerName="payload-processing"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592225    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3da7eac-523d-4d83-8da7-8cbf9b5e583a" containerName="manager"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592231    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592237    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="05587df9-4704-4dc9-a752-16166c5a4f8d" containerName="istio-proxy"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592242    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="811d65a5-42ed-40db-ad04-d539521fbc40" containerName="payload-pre-processing"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592248    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy"
Sep 09 03:38:54.592548 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.592254    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c67ea8-e39e-499c-965a-369909fd1478" containerName="istio-proxy"
Sep 09 03:38:54.595298 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.595282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.599277 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.599245    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 03:38:54.599457 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.599292    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 03:38:54.599457 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.599350    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 03:38:54.599457 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.599418    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-92449\""
Sep 09 03:38:54.606628 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.606605    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-d9jks"]
Sep 09 03:38:54.690618 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.690571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-webhook-cert\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.690618 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.690619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgpp\" (UniqueName: \"kubernetes.io/projected/131ad2b2-8950-482c-94da-d98432a3647d-kube-api-access-shgpp\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.690880 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.690685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-metrics-certs\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.791845 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.791798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-metrics-certs\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.792020 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.791902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-webhook-cert\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.792020 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.791936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgpp\" (UniqueName: \"kubernetes.io/projected/131ad2b2-8950-482c-94da-d98432a3647d-kube-api-access-shgpp\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.794618 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.794593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-webhook-cert\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.794722 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.794593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-metrics-certs\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.800563 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.800538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgpp\" (UniqueName: \"kubernetes.io/projected/131ad2b2-8950-482c-94da-d98432a3647d-kube-api-access-shgpp\") pod \"maas-controller-66bcff4f74-d9jks\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") " pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:54.905597 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:54.905559    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:55.179580 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:55.179495    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-d9jks"]
Sep 09 03:38:55.182063 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:38:55.182034    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131ad2b2_8950_482c_94da_d98432a3647d.slice/crio-f8a85e5bb7cd1bc8b62c65caa29f1c051acca143aee819a40db1cdb834872875 WatchSource:0}: Error finding container f8a85e5bb7cd1bc8b62c65caa29f1c051acca143aee819a40db1cdb834872875: Status 404 returned error can't find the container with id f8a85e5bb7cd1bc8b62c65caa29f1c051acca143aee819a40db1cdb834872875
Sep 09 03:38:55.183238 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:55.183220    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:38:56.040663 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:56.040566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-d9jks" event={"ID":"131ad2b2-8950-482c-94da-d98432a3647d","Type":"ContainerStarted","Data":"29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68"}
Sep 09 03:38:56.040663 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:56.040630    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-d9jks" event={"ID":"131ad2b2-8950-482c-94da-d98432a3647d","Type":"ContainerStarted","Data":"f8a85e5bb7cd1bc8b62c65caa29f1c051acca143aee819a40db1cdb834872875"}
Sep 09 03:38:56.041125 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:56.040750    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:38:56.062537 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:38:56.062481    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66bcff4f74-d9jks" podStartSLOduration=1.5623091279999999 podStartE2EDuration="2.062463262s" totalImagesPullingTime="500.154134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:38:56.061393413 +0000 UTC m=+1985.575475563" watchObservedRunningTime="2026-09-09 03:38:56.062463262 +0000 UTC m=+1985.576545411"
Sep 09 03:39:07.046992 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:07.046959    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:39:16.390878 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.390841    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-d9jks"]
Sep 09 03:39:16.391289 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.391043    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66bcff4f74-d9jks" podUID="131ad2b2-8950-482c-94da-d98432a3647d" containerName="manager" containerID="cri-o://29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68" gracePeriod=30
Sep 09 03:39:16.644574 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.644507    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:39:16.703947 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.703902    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-webhook-cert\") pod \"131ad2b2-8950-482c-94da-d98432a3647d\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") "
Sep 09 03:39:16.704169 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.703995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-metrics-certs\") pod \"131ad2b2-8950-482c-94da-d98432a3647d\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") "
Sep 09 03:39:16.704169 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.704036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shgpp\" (UniqueName: \"kubernetes.io/projected/131ad2b2-8950-482c-94da-d98432a3647d-kube-api-access-shgpp\") pod \"131ad2b2-8950-482c-94da-d98432a3647d\" (UID: \"131ad2b2-8950-482c-94da-d98432a3647d\") "
Sep 09 03:39:16.706186 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.706152    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "131ad2b2-8950-482c-94da-d98432a3647d" (UID: "131ad2b2-8950-482c-94da-d98432a3647d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:39:16.706339 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.706238    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131ad2b2-8950-482c-94da-d98432a3647d-kube-api-access-shgpp" (OuterVolumeSpecName: "kube-api-access-shgpp") pod "131ad2b2-8950-482c-94da-d98432a3647d" (UID: "131ad2b2-8950-482c-94da-d98432a3647d"). InnerVolumeSpecName "kube-api-access-shgpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:39:16.706339 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.706317    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "131ad2b2-8950-482c-94da-d98432a3647d" (UID: "131ad2b2-8950-482c-94da-d98432a3647d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:39:16.804731 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.804692    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-webhook-cert\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:39:16.804731 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.804729    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131ad2b2-8950-482c-94da-d98432a3647d-metrics-certs\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:39:16.804731 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:16.804738    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shgpp\" (UniqueName: \"kubernetes.io/projected/131ad2b2-8950-482c-94da-d98432a3647d-kube-api-access-shgpp\") on node \"ip-10-0-143-233.ec2.internal\" DevicePath \"\""
Sep 09 03:39:17.142672 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.142634    2583 generic.go:356] "Generic (PLEG): container finished" podID="131ad2b2-8950-482c-94da-d98432a3647d" containerID="29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68" exitCode=0
Sep 09 03:39:17.142942 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.142693    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-d9jks"
Sep 09 03:39:17.142942 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.142698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-d9jks" event={"ID":"131ad2b2-8950-482c-94da-d98432a3647d","Type":"ContainerDied","Data":"29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68"}
Sep 09 03:39:17.142942 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.142857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-d9jks" event={"ID":"131ad2b2-8950-482c-94da-d98432a3647d","Type":"ContainerDied","Data":"f8a85e5bb7cd1bc8b62c65caa29f1c051acca143aee819a40db1cdb834872875"}
Sep 09 03:39:17.142942 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.142892    2583 scope.go:117] "RemoveContainer" containerID="29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68"
Sep 09 03:39:17.151683 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.151663    2583 scope.go:117] "RemoveContainer" containerID="29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68"
Sep 09 03:39:17.152027 ip-10-0-143-233 kubenswrapper[2583]: E0909 03:39:17.152004    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68\": container with ID starting with 29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68 not found: ID does not exist" containerID="29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68"
Sep 09 03:39:17.152106 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.152042    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68"} err="failed to get container status \"29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68\": rpc error: code = NotFound desc = could not find container \"29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68\": container with ID starting with 29b67622e64cb6a34694de55a43853d7beae47f8024e944f18c6228d11a51c68 not found: ID does not exist"
Sep 09 03:39:17.161790 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.161766    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-d9jks"]
Sep 09 03:39:17.167160 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.167141    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-d9jks"]
Sep 09 03:39:17.989067 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.989030    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66bcff4f74-6szns"]
Sep 09 03:39:17.990177 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.990152    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy"
Sep 09 03:39:17.990307 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.990187    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy"
Sep 09 03:39:17.990307 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.990246    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131ad2b2-8950-482c-94da-d98432a3647d" containerName="manager"
Sep 09 03:39:17.990307 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.990255    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="131ad2b2-8950-482c-94da-d98432a3647d" containerName="manager"
Sep 09 03:39:17.990549 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.990535    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="100bc079-5a46-413b-8f23-0d89e2b6ea94" containerName="istio-proxy"
Sep 09 03:39:17.990617 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.990560    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="131ad2b2-8950-482c-94da-d98432a3647d" containerName="manager"
Sep 09 03:39:17.998262 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:17.998234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.002030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.002004    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-92449\""
Sep 09 03:39:18.002030 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.002024    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 03:39:18.002231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.002039    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 03:39:18.002231 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.002044    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 03:39:18.008064 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.008040    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-6szns"]
Sep 09 03:39:18.117613 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.117579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2088cf66-f974-4a7e-8b99-636349257b06-webhook-cert\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.117800 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.117622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2088cf66-f974-4a7e-8b99-636349257b06-metrics-certs\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.117932 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.117905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvfs7\" (UniqueName: \"kubernetes.io/projected/2088cf66-f974-4a7e-8b99-636349257b06-kube-api-access-lvfs7\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.218801 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.218769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvfs7\" (UniqueName: \"kubernetes.io/projected/2088cf66-f974-4a7e-8b99-636349257b06-kube-api-access-lvfs7\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.219077 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.218846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2088cf66-f974-4a7e-8b99-636349257b06-webhook-cert\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.219077 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.218884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2088cf66-f974-4a7e-8b99-636349257b06-metrics-certs\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.221543 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.221517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2088cf66-f974-4a7e-8b99-636349257b06-metrics-certs\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.221636 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.221550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2088cf66-f974-4a7e-8b99-636349257b06-webhook-cert\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.228270 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.228248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvfs7\" (UniqueName: \"kubernetes.io/projected/2088cf66-f974-4a7e-8b99-636349257b06-kube-api-access-lvfs7\") pod \"maas-controller-66bcff4f74-6szns\" (UID: \"2088cf66-f974-4a7e-8b99-636349257b06\") " pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.310655 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.310558    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:18.586618 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:18.586593    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66bcff4f74-6szns"]
Sep 09 03:39:18.588467 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:39:18.588436    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2088cf66_f974_4a7e_8b99_636349257b06.slice/crio-24b239906eca6bb9b63f12e02a193c308429cd83e92706de2a72161a8d41a88e WatchSource:0}: Error finding container 24b239906eca6bb9b63f12e02a193c308429cd83e92706de2a72161a8d41a88e: Status 404 returned error can't find the container with id 24b239906eca6bb9b63f12e02a193c308429cd83e92706de2a72161a8d41a88e
Sep 09 03:39:19.006953 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:19.006912    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="131ad2b2-8950-482c-94da-d98432a3647d" path="/var/lib/kubelet/pods/131ad2b2-8950-482c-94da-d98432a3647d/volumes"
Sep 09 03:39:19.154891 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:19.154773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-6szns" event={"ID":"2088cf66-f974-4a7e-8b99-636349257b06","Type":"ContainerStarted","Data":"75c4bfd86903f50bb524d1c62b15cfa84be24444a39e149c8e61595d7bed0b95"}
Sep 09 03:39:19.154891 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:19.154854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66bcff4f74-6szns" event={"ID":"2088cf66-f974-4a7e-8b99-636349257b06","Type":"ContainerStarted","Data":"24b239906eca6bb9b63f12e02a193c308429cd83e92706de2a72161a8d41a88e"}
Sep 09 03:39:19.155098 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:19.154987    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:39:19.179562 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:19.179492    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66bcff4f74-6szns" podStartSLOduration=1.8664407650000001 podStartE2EDuration="2.179469794s" totalImagesPullingTime="313.029029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:39:19.178179306 +0000 UTC m=+2008.692261454" watchObservedRunningTime="2026-09-09 03:39:19.179469794 +0000 UTC m=+2008.693551944"
Sep 09 03:39:30.162413 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:39:30.162379    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66bcff4f74-6szns"
Sep 09 03:40:51.084199 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:40:51.084163    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:40:51.089691 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:40:51.089664    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:40:51.090782 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:40:51.090759    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:40:51.095639 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:40:51.095621    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-94lfn_6b7fca20-9b25-4a26-a49d-3089acb2966e/ovn-acl-logging/0.log"
Sep 09 03:43:33.169957 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:33.169883    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fbbdbb45d-2prsm_d940a15e-5591-40b6-9049-9e81a6f69920/authorino/0.log"
Sep 09 03:43:39.106809 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:39.106777    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66bcff4f74-6szns_2088cf66-f974-4a7e-8b99-636349257b06/manager/0.log"
Sep 09 03:43:39.222325 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:39.222295    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zxjpc_8977b470-c5fe-48f6-bf98-98f2c2539a1b/server/0.log"
Sep 09 03:43:39.355019 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:39.354986    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-566b4_7d060b18-4360-4ba2-b8ac-320431078102/manager/0.log"
Sep 09 03:43:39.771157 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:39.771132    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-t7cg2_b292da09-4620-4521-871c-4a6e9a8dd3e8/copy-manifests/0.log"
Sep 09 03:43:39.777267 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:39.777251    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-t7cg2_b292da09-4620-4521-871c-4a6e9a8dd3e8/manager/0.log"
Sep 09 03:43:41.321888 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:41.321851    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fbbdbb45d-2prsm_d940a15e-5591-40b6-9049-9e81a6f69920/authorino/0.log"
Sep 09 03:43:42.479691 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:42.479664    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn_d0f848d1-662d-4c7a-83a8-a44975d79e72/istio-proxy/0.log"
Sep 09 03:43:42.837546 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:42.837467    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8d5f99fb-794r4_25b63d06-0e54-47c5-bfef-8afebf0b5fa3/kube-auth-proxy/0.log"
Sep 09 03:43:43.688361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:43.688330    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-65jl5_b73e19ef-9123-4c0c-801d-d0b287c54767/main/0.log"
Sep 09 03:43:43.935763 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:43.935732    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-qmlrs_ed74d6f5-7ce8-44f5-a8e9-eb6ae3e51dd5/main/0.log"
Sep 09 03:43:44.058929 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:44.058852    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8xs9k_d44558e4-6f32-497e-afb3-e79a58277da2/main/0.log"
Sep 09 03:43:44.416539 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:44.416507    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-mvldf_f982dadf-881e-4d85-b5fe-88e90bc22a0b/main/0.log"
Sep 09 03:43:53.259536 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:53.259505    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g698z_e353f605-9075-414d-aa37-b8b53f094c02/global-pull-secret-syncer/0.log"
Sep 09 03:43:53.384268 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:53.384235    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x4mml_4fae6eb0-381c-4cc5-9098-8c95abeca3ba/konnectivity-agent/0.log"
Sep 09 03:43:53.464303 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:53.464272    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-233.ec2.internal_3f4776863423ca28036bea3b8e2635ca/haproxy/0.log"
Sep 09 03:43:58.382600 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:43:58.382559    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fbbdbb45d-2prsm_d940a15e-5591-40b6-9049-9e81a6f69920/authorino/0.log"
Sep 09 03:44:00.341743 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:00.341710    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-dldlp_89139c70-916b-4714-a28e-80202823aa6d/cluster-monitoring-operator/0.log"
Sep 09 03:44:00.465692 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:00.465666    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-gn28j_9c67e593-d7dc-401b-8e6a-61a92a35100e/monitoring-plugin/0.log"
Sep 09 03:44:00.500516 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:00.500488    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-746hs_db96c9f0-a549-47ee-ba38-fa42f5296d55/node-exporter/0.log"
Sep 09 03:44:00.526983 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:00.526960    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-746hs_db96c9f0-a549-47ee-ba38-fa42f5296d55/kube-rbac-proxy/0.log"
Sep 09 03:44:00.553738 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:00.553712    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-746hs_db96c9f0-a549-47ee-ba38-fa42f5296d55/init-textfile/0.log"
Sep 09 03:44:01.119796 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:01.119749    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-z9vgj_a88b5550-e0ea-4fdc-8e53-706f27af9828/prometheus-operator-admission-webhook/0.log"
Sep 09 03:44:01.160928 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:01.160900    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d8c95897f-7rfjg_735ee480-c750-4fc9-baa2-93e5a6ae41ff/telemeter-client/0.log"
Sep 09 03:44:01.185653 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:01.185623    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d8c95897f-7rfjg_735ee480-c750-4fc9-baa2-93e5a6ae41ff/reload/0.log"
Sep 09 03:44:01.206958 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:01.206934    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d8c95897f-7rfjg_735ee480-c750-4fc9-baa2-93e5a6ae41ff/kube-rbac-proxy/0.log"
Sep 09 03:44:03.448483 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:03.448449    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/1.log"
Sep 09 03:44:03.453293 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:03.453265    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-l8xmz_d66383af-0733-4820-b76b-3c4ca43d413b/console-operator/2.log"
Sep 09 03:44:03.942626 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:03.942590    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"]
Sep 09 03:44:03.946253 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:03.946232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:03.948735 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:03.948713    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nn8m\"/\"default-dockercfg-k9ktg\""
Sep 09 03:44:03.948900 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:03.948732    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nn8m\"/\"kube-root-ca.crt\""
Sep 09 03:44:03.948900 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:03.948719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nn8m\"/\"openshift-service-ca.crt\""
Sep 09 03:44:03.953489 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:03.953471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"]
Sep 09 03:44:04.023291 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.023257    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-dp6tq_a5fce05f-0b08-457d-935a-b6b30a5c83fc/download-server/0.log"
Sep 09 03:44:04.116636 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.116594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-sys\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.116806 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.116657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-podres\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.116806 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.116741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-lib-modules\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.116922 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.116802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d98hd\" (UniqueName: \"kubernetes.io/projected/4bfca206-3bdb-4a3d-9734-528cc44c497a-kube-api-access-d98hd\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.116922 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.116886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-proc\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-lib-modules\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218092 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98hd\" (UniqueName: \"kubernetes.io/projected/4bfca206-3bdb-4a3d-9734-528cc44c497a-kube-api-access-d98hd\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-proc\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-sys\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-sys\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-proc\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218251 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218212    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-lib-modules\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218453 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-podres\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.218453 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.218372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bfca206-3bdb-4a3d-9734-528cc44c497a-podres\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.226575 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.226549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98hd\" (UniqueName: \"kubernetes.io/projected/4bfca206-3bdb-4a3d-9734-528cc44c497a-kube-api-access-d98hd\") pod \"perf-node-gather-daemonset-kgpsj\" (UID: \"4bfca206-3bdb-4a3d-9734-528cc44c497a\") " pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.257409 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.257382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:04.529532 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.529507    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"]
Sep 09 03:44:04.531650 ip-10-0-143-233 kubenswrapper[2583]: W0909 03:44:04.531624    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4bfca206_3bdb_4a3d_9734_528cc44c497a.slice/crio-1899117a0043af1a8f36266716861393690521817494b35315dee053137f153a WatchSource:0}: Error finding container 1899117a0043af1a8f36266716861393690521817494b35315dee053137f153a: Status 404 returned error can't find the container with id 1899117a0043af1a8f36266716861393690521817494b35315dee053137f153a
Sep 09 03:44:04.740196 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:04.740167    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-sgmx8_019cb9a5-1c1a-474f-b098-85ce52ff122b/volume-data-source-validator/0.log"
Sep 09 03:44:05.524279 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:05.524247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj" event={"ID":"4bfca206-3bdb-4a3d-9734-528cc44c497a","Type":"ContainerStarted","Data":"d1fb54995bf875b428c53071ebd49460c323c7e511e5c5f845abd98a79427af5"}
Sep 09 03:44:05.524460 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:05.524297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj" event={"ID":"4bfca206-3bdb-4a3d-9734-528cc44c497a","Type":"ContainerStarted","Data":"1899117a0043af1a8f36266716861393690521817494b35315dee053137f153a"}
Sep 09 03:44:05.524460 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:05.524339    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:05.542573 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:05.542523    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj" podStartSLOduration=2.542509186 podStartE2EDuration="2.542509186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:44:05.540069542 +0000 UTC m=+2295.054151703" watchObservedRunningTime="2026-09-09 03:44:05.542509186 +0000 UTC m=+2295.056591335"
Sep 09 03:44:05.928145 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:05.928108    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rg75s_b72822b1-64d7-48a4-9101-4b8dcbb6ef09/dns/0.log"
Sep 09 03:44:05.951133 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:05.951064    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rg75s_b72822b1-64d7-48a4-9101-4b8dcbb6ef09/kube-rbac-proxy/0.log"
Sep 09 03:44:05.976746 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:05.976724    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9zhgm_742fbb48-0057-4d26-9b3c-2493b94a952e/dns-node-resolver/0.log"
Sep 09 03:44:06.667154 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:06.667091    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54bf464b6f-7wmhs_f725e7d0-9c83-4232-92d7-dab1bf647b30/registry/0.log"
Sep 09 03:44:06.675908 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:06.675850    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54bf464b6f-7wmhs_f725e7d0-9c83-4232-92d7-dab1bf647b30/registry/1.log"
Sep 09 03:44:06.700756 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:06.700729    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2p8lk_9b96da89-37da-4ea3-a3f6-33282f05e059/node-ca/0.log"
Sep 09 03:44:07.687523 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:07.687495    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn_d0f848d1-662d-4c7a-83a8-a44975d79e72/istio-proxy/0.log"
Sep 09 03:44:07.964554 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:07.964443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8d5f99fb-794r4_25b63d06-0e54-47c5-bfef-8afebf0b5fa3/kube-auth-proxy/0.log"
Sep 09 03:44:08.994411 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:08.994376    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dvljc_dae8f832-f923-4354-9649-b1ddc523ebf8/serve-healthcheck-canary/0.log"
Sep 09 03:44:09.496312 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:09.496279    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vt567_c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1/insights-operator/0.log"
Sep 09 03:44:09.496818 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:09.496793    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vt567_c54ade31-ffe0-4cc4-9a3c-b02b5a7523b1/insights-operator/1.log"
Sep 09 03:44:09.651670 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:09.651639    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncfxg_c6c32567-4f89-4461-b0a4-81621726808a/kube-rbac-proxy/0.log"
Sep 09 03:44:09.672361 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:09.672332    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncfxg_c6c32567-4f89-4461-b0a4-81621726808a/exporter/0.log"
Sep 09 03:44:09.695223 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:09.695196    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncfxg_c6c32567-4f89-4461-b0a4-81621726808a/extractor/0.log"
Sep 09 03:44:11.539307 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:11.539279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nn8m/perf-node-gather-daemonset-kgpsj"
Sep 09 03:44:12.104934 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:12.104903    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66bcff4f74-6szns_2088cf66-f974-4a7e-8b99-636349257b06/manager/0.log"
Sep 09 03:44:12.125595 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:12.125574    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zxjpc_8977b470-c5fe-48f6-bf98-98f2c2539a1b/server/0.log"
Sep 09 03:44:12.197570 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:12.197536    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-566b4_7d060b18-4360-4ba2-b8ac-320431078102/manager/0.log"
Sep 09 03:44:12.392360 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:12.392332    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-t7cg2_b292da09-4620-4521-871c-4a6e9a8dd3e8/manager/0.log"
Sep 09 03:44:12.416630 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:12.416598    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856874fcb4-t7cg2_b292da09-4620-4521-871c-4a6e9a8dd3e8/copy-manifests/0.log"
Sep 09 03:44:13.927270 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:13.927237    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f6f9bc9b9-xp24w_8018038d-bd20-425b-9b38-ed0a2abd8554/manager/0.log"
Sep 09 03:44:13.955712 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:13.955672    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-575vx_409bbb80-c5d0-4f1a-8fb9-e2497463f18d/openshift-lws-operator/0.log"
Sep 09 03:44:14.921035 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:14.921010    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn_d0f848d1-662d-4c7a-83a8-a44975d79e72/istio-proxy/0.log"
Sep 09 03:44:15.160623 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:15.160589    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8d5f99fb-794r4_25b63d06-0e54-47c5-bfef-8afebf0b5fa3/kube-auth-proxy/0.log"
Sep 09 03:44:16.943349 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:16.943322    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-f2gdn_d0f848d1-662d-4c7a-83a8-a44975d79e72/istio-proxy/0.log"
Sep 09 03:44:17.218823 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:17.218744    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8d5f99fb-794r4_25b63d06-0e54-47c5-bfef-8afebf0b5fa3/kube-auth-proxy/0.log"
Sep 09 03:44:19.228058 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:19.228008    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-78cwv_11b8db99-a950-4d63-befb-1ce9d91be8a3/migrator/0.log"
Sep 09 03:44:19.248596 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:19.248572    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-78cwv_11b8db99-a950-4d63-befb-1ce9d91be8a3/graceful-termination/0.log"
Sep 09 03:44:19.596924 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:19.596824    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-k6749_0a215f92-12cb-448a-b375-5a5b668a0998/kube-storage-version-migrator-operator/1.log"
Sep 09 03:44:19.597673 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:19.597650    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-k6749_0a215f92-12cb-448a-b375-5a5b668a0998/kube-storage-version-migrator-operator/0.log"
Sep 09 03:44:20.793915 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:20.793886    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w82s_7d565593-7fbb-41ef-8856-8b6dfa88a62c/kube-multus-additional-cni-plugins/0.log"
Sep 09 03:44:20.814344 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:20.814319    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w82s_7d565593-7fbb-41ef-8856-8b6dfa88a62c/egress-router-binary-copy/0.log"
Sep 09 03:44:20.835081 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:20.835056    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w82s_7d565593-7fbb-41ef-8856-8b6dfa88a62c/cni-plugins/0.log"
Sep 09 03:44:20.860784 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:20.860763    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w82s_7d565593-7fbb-41ef-8856-8b6dfa88a62c/bond-cni-plugin/0.log"
Sep 09 03:44:20.880652 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:20.880632    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w82s_7d565593-7fbb-41ef-8856-8b6dfa88a62c/routeoverride-cni/0.log"
Sep 09 03:44:20.899674 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:20.899639    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w82s_7d565593-7fbb-41ef-8856-8b6dfa88a62c/whereabouts-cni-bincopy/0.log"
Sep 09 03:44:20.919773 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:20.919748    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w82s_7d565593-7fbb-41ef-8856-8b6dfa88a62c/whereabouts-cni/0.log"
Sep 09 03:44:21.130761 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:21.130727    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bccdk_39f5888a-646d-4f07-9073-aebd7f316182/kube-multus/0.log"
Sep 09 03:44:21.321936 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:21.321903    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zqzm9_734355ce-8feb-4305-8872-543d37a47fc4/network-metrics-daemon/0.log"
Sep 09 03:44:21.338981 ip-10-0-143-233 kubenswrapper[2583]: I0909 03:44:21.338957    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zqzm9_734355ce-8feb-4305-8872-543d37a47fc4/kube-rbac-proxy/0.log"